[#11] oriondb create neondb instance and persistence layer

[diyakadakia]

Overview

Created files schema.py, exceptions.py, and repository.py, plus write_test_orion.py.
schema.py: defines the 'data' table on Neon, mirroring the DataPoint model from our previous ticket
exceptions.py: custom error types for connection/schema/validation failures.
repository.py: generic, reusable write-only repository. Validates records before writing so invalid data can't get in, and safely skips duplicates if the same data is written twice.

Technical Description

Set up a Neon Postgres instance, defined the table schema to match the ingest model from ticket 1, and built a write-only persistence layer with a generic base class so future tables can reuse the same write logic. Validation happens before any DB connection is opened, so an invalid record never reaches the database.

How Has This Been Tested?

Ran setup_orion.py once to create the table on the live Neon instance. Ran write_test_orion.py against the real Penelope + Neon databases
which:

• pulls a real window of data from Penelope
• writes it to Orion and confirms the write succeeded
• reads it back and verifies it matches exactly what came from Penelope
• attempts to write invalid records and confirms they're rejected before touching the database

Ticket Review

Answer the following:

1. What went well
   Successfully completed the ticket!

2. What was difficult
   This was our first time using NeonDB so we had to learn what it was first. We also could not pull all the data so we had to query it in order to test it.

3. If you could start over what would you do differently
   Nothing — this was a good start at using Neon and building out the persistence layer

[chelseakwan]

For some reason it won't let me hyperlink the issue so I'm just pasting it here:
#11

Overview
Provide a brief summary of the changes you made and files touched.
Revised repository.py: updated to be more in line with scope of project and to follow new numpy arrays formatting
Revise write_orion_test.py: Revised testing to be in line with arrays

Technical Description
How did you achieve this goal. Explain briefly how your change achives the ticket's ask.
Reduced respository.py to be more simple and less over-engineered. Now properly writes data into Orion DB via arrays.

How Has This Been Tested?
Describe your steps for testing this work. Include screenshots or unit test files.
Used write_orion_test.py to verify lines were being properly written into new database

Ticket Review
Answer the following:

What went well
Successful fixed ticket
What was difficult
Wasn't too difficult, a lot of the code was simplified when we switch everything to arrays
If you could start over what would you do differently
Ensure that everything is written to the scale of the project!

[mdeekshita]

Looks good and tests work!

[henrybednarz]

This leaves a hanging connection if there the caller doesn't pull every record from the iterator. E.g. something fails downstream. You can get around this by having the server hold a cursor of your spot instead of leaving an open connection.

I made a fix for you:

def get_all_paginated(self, batch_size: int = 5000):
    if batch_size <= 0:
        raise ValueError("batch_size must be positive")

    stmt = select(self.data_table).order_by(self.data_table.c.time)

    try:
        with self.engine.connect().execution_options(yield_per=batch_size) as conn:
            result = conn.execute(stmt)
            for partition in result.partitions():
                yield self._to_array(partition)
    except OperationalError as exc:
        raise PenelopeConnectionError(
            f"Lost connection to PenelopeDB at {self.engine.url.host}:"
            f"{self.engine.url.port} during a streamed read. "
            "Check that the VPN is still connected"
        ) from exc
    except SQLAlchemyError as exc:
        raise PenelopeSchemaError(
            "Streamed query against PenelopeDB failed."
        ) from exc

[henrybednarz]

Other than that looks good. Just resolve the merge conflict then ill approve