--Project Overview-- Ember is a modern platform (Node.js backend, Vercel, Supabase/Postgres, React frontend) serving investment professionals with tools to streamline research and decision-making. We’re embarking on a new AI agent-powered news feed feature that will help financial analysts efficiently track relevant updates in their specific industry sub-sectors. Think of it as a smart, personalized news assistant: it will ingest a “firehose” of news and distill it into the insights that matter most for each user’s niche. We have a clear vision for this system’s capabilities and need a seasoned engineer to help bring it to life. --Key System Goals & Features:-- Agentic News Aggregation: Design an autonomous agent system that pulls in news from multiple sources – including earnings call transcripts, paywalled financial news sites, social media posts, industry blogs, etc. – ensuring comprehensive coverage of each sector. Per-Agent Customization: Each agent focuses on a specific sector or topic. The system should support configuring an agent’s focus (e.g. Tech Sector, Healthcare), with the ability to whitelist or blacklist sources and incorporate user feedback to refine relevance over time. In other words, the feed should get smarter and more personalized the more it’s used. Persistent Memory: Implement storage of collected news content and context so that each agent has “memory.” This allows agents to recognize what’s been seen before, avoid repetition, and provide context-aware summaries. Storing this data (likely in a database or knowledge base) will enable the agent to recall past events when generating new reports. Automated Recaps: The agents will autonomously generate structured summaries (daily and weekly digests) of the most important news for their sector. An analyst can receive, say, a morning briefing and a weekly round-up tailored to what they care about, instead of sifting through dozens of articles. Future-Proof Architecture: While the initial goal is a one-way feed from agents to user, we want the architecture to allow future inter-agent communication. Down the line, agents might share insights with each other (for example, a Tech agent alerting a Finance agent about a cross-industry development). Designing with this in mind now will make expansions easier later. We’re intentionally not prescribing exact implementation details – you will have the freedom to shape the solution (choice of libraries, frameworks, AI integration methods, etc.) as long as the system meets these goals. Our focus is on outcomes: a robust, intelligent news feed that delights users. The project will start with an MVP build (~8–12 weeks) to get a working system in place. If all goes well, there’s a strong possibility for a longer-term collaboration to keep improving and scaling this product. --Key Responsibilities-- In this role, you will take the lead in engineering this AI-driven news feed from the ground up. Your responsibilities will include: Architect & Develop the Agent System: Design and implement the core agent-based news feed system on Ember’s platform. This includes defining how agents are structured, how they ingest and process information, and how results are delivered to the frontend. You’ll make high-level architectural decisions (data models, service design, integration points) and then get hands-on in building it. Multi-Source Data Ingestion: Create robust pipelines to ingest content from diverse sources. Many sources will have public APIs, but others won’t – you’ll need to implement custom web scrapers or crawlers for sites that lack APIs or have anti-scraping measures. The goal is to reliably pull in articles, transcripts, social posts, etc., normalizing them for the agent to analyze. (Some sources may require creative solutions due to dynamic content loading or paywalls – part of the challenge!) Intelligent Filtering & Deduplication: Develop logic to filter out noise and duplicate information across sources. If the same story appears in multiple outlets, the system should recognize it and avoid bombarding the user. This will likely involve comparing content for similarity and using meta-data. You’ll also incorporate smart filtering rules (for example, removing irrelevant clickbait or low-value content) and plug in user feedback loops – e.g. if a user thumbs-down certain types of stories, the agent should learn and adjust. Over time, the feed should learn the user’s preferences and surface more of what’s useful to them. LLM Integration (Model-Agnostic): Integrate large language models to power the agents’ understanding and summarization capabilities. The system should be model-agnostic – initially we might use OpenAI’s APIs (GPT-4/3.5), but we want the flexibility to swap in other providers like Anthropic or open-source models down the road. You’ll architect the integration in a modular way (e.g., via an abstraction layer or service) so we’re not locked into a single AI model. The LLM will likely be used to summarize batches of articles and possibly to decide relevance, so experience with prompt design and handling LLM output will be helpful. Agent Output & Summaries: Ensure each agent can produce clean, structured output for users. This includes generating the daily/weekly recap summaries and possibly real-time alerts for breaking news. You’ll work on how these summaries are formatted and delivered (likely via our React frontend, maybe email notifications in the future). The output should be accurate and concise – part of your job is figuring out how to prompt the model or post-process results to achieve this. Data Storage & Management: Set up the necessary data storage for this feature – both for incoming content (raw articles, transcripts, etc.) and the agent’s processed results. We use Supabase (Postgres) currently, so you’ll design tables or utilize Supabase’s features to store content and metadata. This stored knowledge will serve as the agent’s “long-term memory” and also allow auditability (e.g., to trace why an agent made a certain summary, we can reference the source data). Scalability and security of this data are important; as we ingest potentially hundreds of articles daily, the solution should remain performant. Collaboration & Iteration: Work closely with Ember’s founder and product/design team to refine requirements and iterate on the solution. This is a highly collaborative project – for example, you might need to adjust how an agent’s configuration is exposed in the UI or tweak summarization based on user beta testing. You should be comfortable communicating technical trade-offs to non-engineers and incorporating feedback quickly. Quality, Testing & Deployment: Given the nature of news content and AI, testing will be an ongoing responsibility. You’ll implement checks to ensure the scraping and ingestion are working (no silent failures), possibly write unit/integration tests for the pipeline, and help devise ways to evaluate the quality of the AI summaries (e.g. spot-checking outputs). We are hosted on Vercel, so you’ll also be involved in deploying any backend functions or services you build and monitoring their performance. Security & Future Compliance: As we build, keep in mind enterprise readiness. In the long run, our target customers may require compliance like SOC 2, so the system should be designed with good security practices from day one. This might include proper handling of API keys and secrets, secure data storage (especially if any news content might be sensitive or behind paywalls), logging and monitoring for the new features, etc. We don’t need full SOC 2 controls in the MVP, but we want a foundation that won’t require a complete rewrite to get there. Drive the MVP and Beyond: Take ownership of delivering a working MVP of this news feed system within ~8–12 weeks. This includes planning the work, executing on core features, and adapting as requirements evolve. After the MVP, there’s potential to continue iterating on advanced features (like multi-agent interactions, more personalization, scaling to more users, etc.). You would play a key role in that continued development if there’s a mutual fit. In short, you’re not just writing code – you’re leading the technical effort to make this project a success. To help increase context, here is an AI generated UX Mock-up of the screens we want to enable: https://claude.ai/public/artifacts/38414b21-c877-4f8e-8b6d-adbe7528d9e2