Fix AI Agent documentation and add API endpoints for load balancers

[AnkanSaha]

This pull request introduces several improvements to the EdgeBalancer web client, focusing on enhanced documentation, SEO/metadata, and user experience. The most significant changes include the addition of an About page, expanded API and platform documentation, improved SEO (including Open Graph and structured data), and better error handling for 404 pages. There are also updates to navigation and robots/sitemap configuration to support these enhancements.

Documentation & Platform Information:

• Added detailed "When to use", "When NOT to use", and "How EdgeBalancer works for agents" sections, as well as full API endpoint documentation to llms.txt and llms-full.txt. This clarifies platform capabilities, limitations, and available API operations for both users and AI agents. [1] [2]

User Experience Improvements:

• Introduced a new /about page (page.tsx) describing EdgeBalancer’s mission, features, and pricing, and added a link to it in the site footer and sitemap. [1] [2] [3]
• Implemented a custom 404 "Not Found" page with navigation options to return home or sign in, improving user guidance on invalid URLs.

SEO & Metadata Enhancements:

• Updated Open Graph image URLs in the main layout for better social sharing previews. [1] [2]
• Added a new OrganizationSchema component for structured data, improving search engine understanding of the site’s organization. [1] [2] [3]
• Expanded the robots.txt rules to explicitly allow major AI and web crawlers, increasing discoverability.

Navigation & Discoverability:

• Added links to API documentation (/openapi.json) in both the landing page and primary navigation, making it easier for developers to find integration resources. [1] [2]

Technical Middleware:

• Introduced a simple middleware to set the Vary: Accept header on all responses, improving content negotiation and caching behavior.