🚧🚨 Please note that Cyyrus is currently in active development as we prepare for its public release 🚧🚨
World has plenty of data, but most of it is trapped in formats that are difficult to utilize. We’re talking about messy relational databases, unstructured text, audio, video, even the latent space of LLMs. It's not a goldmine; it's a landfill. And we're spending millions trying to clean it up.
DataOps today looks a mix of ClickOps, CryOps and PrayOps. You're either clicking your life away, crying over broken pipelines, or praying your dataset doesn't implode. Fun times, right?
Cyyrus aims to do for datasets, what Terraform did for Infrastructure. It transforms unstructured data into usable datasets.
| Task | Description | Status | Future Work |
|---|---|---|---|
| Parsing | Parse unstructured data sources (PDF, images) | Beta | Add support for more document types, audio, and video |
| Generation | Generate data using Language Models | Added OpenAI integration | Implement deterministic and rule-based generators |
| Extraction | Seed data from existing datasets (CSV, JSON, etc.) | In development | Coming Soon |
| Scraping | Collect and seed data from web sources | In development | Coming Soon |
| Labelling | Enrich image datasets for detection, classification, and segmentation | To Be Determined | Coming Soon |
Cyyrus does't make assumptions about your data because it doesn't have to. It handles it all. You can create datasets by composing these tasks into one another, enabling declarative dataset generation pipelines. Think of them like Lego bricks.
Sure but doesn't X do this already? Yeah, maybe. The market is saturated with products like these but these tools don't solve data silos; they create new ones.
Cyyrus doesn't bundle an analytics product, it plans to charge for transformations and checkpoints, not data storage so it benefits with data movement. Put simply, the more your data moves, the happier we are. Not the other way round.
We've put together a neat documentation. Check them out (Shoutout to our hyper-cafinated devs prompting cursor day in and out :D).
We built Cyyrus so that it doesn’t need a manual to operate, but here’s one anyway. Let's get you upto speed. You'd be a pro in no time.
Alright, baby steps. Install Cyyrus with pip:
pip install cyyrusWith Cyyrus, you can configure your schema using a simple YAML file. This YAML defines the structure, properties, and types of the dataset. Think of it as poor man's HCL.
-
Define Your Tasks: First step is to define the tasks that will be used in the dataset generation process. These tasks can include parsing, extraction, or something else.
tasks: # Define the invoice parsing task invoice_parsing: task_type: parsing task_properties: directory: experimental/sample file_type: pdf max_depth: 5 parsed_format: base64 # Define the customer info extraction task extract_customer_info: task_type: generation task_properties: model: gpt-4o-mini prompt: Extract customer info from the given invoice response_format: customer_info api_key: $OPENAI_API_KEY
-
Define Your Types: Once the tasks are defined, the next step is to define the types that will be used in the dataset. These types can include objects, arrays, or any other data type.
types: customer_info: type: object properties: customer_name: type: string # What do we call this hooman? customer_address: type: string # Where do we find them? invoice_id: type: string # The unique identifier for their financial pain total_amount: type: float # How much do they owe us for our digital crack?
-
Define Your Columns: Columns are the attributes of the dataset.
columns: # Define the parsed invoice column parsed_invoice: task_id: invoice_parsing # Associate a task_id with the column # Define the customer info column customer_info: task_id: extract_customer_info task_input: [parsed_invoice] # Define the input for the task. # Define the invoice items column invoice_items: task_id: extract_invoice_items task_input: [parsed_invoice] # Define the invoice qna column invoice_qna: task_id: create_invoice_qna task_input: [invoice_items, customer_info] # Ensures the task is executed after the task_input is available
-
Define Your Datasets: Datasets are defined by the metadata, splits, attributes, and shuffle properties.
# Define the properties of the dataset dataset: # Define the metadata of the dataset metadata: name: Invoice Dataset description: Dataset containing the invoice data tags: [invoice, financial, document] license: CC-BY-NC-SA languages: [en] # Define how to shuffle the dataset shuffle: seed: 42 # Define the splits of the dataset splits: train: 0.8 test: 0.2 seed: 42 # Define the attributes of the dataset attributes: required_columns: [invoice_items, customer_info] unique_columns: [] flatten_columns: [invoice_items, invoice_qna] exclude_columns: [parsed_invoice] nulls: include
Now comes the exciting part. With our schema in place, we’re ready to generate data.
cyrus run --schema-path path/to/your/schema.yaml --env-path path/to/your/.env
Cyrus processes each column defined in your schema, handling dependencies, types, error cases, and one-to-many mappings. You’ll see progress bars and logs for each step:
2024-08-26 16:01:14,096 - cyrus.composer.core - INFO - Preparing column: parsed_invoice
2024-08-26 16:01:14,097 - cyrus.composer.core - INFO - Executing task: TaskType.PARSING
100%|█████████████████████████████████████████████████| 1/1 [00:08<00:00, 8.09s/it]
2024-08-26 16:01:22,191 - cyrus.composer.core - INFO - Preparing column: customer_info
2024-08-26 16:01:22,191 - cyrus.composer.core - INFO - Executing task: TaskType.GENERATION
100%|█████████████████████████████████████████████████| 11/11 [00:44<00:00, 4.03s/it]
...
And just like that you unf*ucked unstructured data. Pretty, ain't it?
But we’re not done yet! After generation, you’ll have the option to export your dataset:
Ready to export the dataset? [y/N]: y
Enter the export directory [/Users/Code/cyrus]: export
Enter the export format (huggingface, json, csv, pickle, parquet) [huggingface]: json
Enter a name for your dataset (How about: Pierce_Macadamia ?) [Pierce_Macadamia]: invoice
Choose your flavor - JSON, CSV, pickle, parquet - Cyrus has got you covered.
Cyrus is best buddies with Hugging Face, and it’s ready to help you publish your dataset.
Do you want to publish the dataset? [Y/N]: y
HF TOKEN found in environment. Use 'hf_PT...NFTJu'? [Y/N]: y
Enter the repository identifier: wizenheimer/invoice-dataset
Keep the dataset private? [Y/N]: y
Cyrus handles the upload process, creating the necessary formats and uploading to Hugging Face:
2024-08-26 16:05:35,604 - cyrus.composer.core - INFO - Publishing dataset to Hugging Face: wizenheimer/invoice-dataset
Creating parquet from Arrow format: 100%|████████████████| 1/1 [00:00<00:00, 152.07ba/s]
Uploading the dataset shards: 100%|███████████████████| 1/1 [00:03<00:00, 3.04s/it]
Creating parquet from Arrow format: 100%|████████████████| 1/1 [00:00<00:00, 255.36ba/s]
Uploading the dataset shards: 100%|███████████████████| 1/1 [00:01<00:00, 1.60s/it]
2024-08-26 16:05:42,223 - cyrus.composer.core - INFO - Dataset successfully published to wizenheimer/invoice-dataset
2024-08-26 16:05:42,224 - cyrus.cli.main - INFO - Published dataset to None. Happy sharing!
2024-08-26 16:05:42,224 - cyrus.cli.main - INFO - Dataset published successfully!
And there you have it, folks! your dataset is generated, exported, and published to Hugging Face
We've put together a neat documentation. Check out docs.
Additionally example directory contains schema examples, sample datasets generated using cyyrus and even a Jupyter notebook to help you get started with Cyyrus.
examples
├── README.md
├── exports
│ ├── doclaynet_bench
│ │ └── result.jsonl
│ ├── funsd_layoutlmv3
│ │ └── result.jsonl
│ ├── invoices_receipts_ocr_v1
│ │ └── result.jsonl
│ └── layoutlm_resume_data
│ └── result.jsonl
├── notebook
│ └── schema_cookbook.ipynb
└── schema
├── annual_report_to_markdown.yaml
├── doclaynet_bench.yaml
├── extract_info_from_invoice.yaml
├── funsd_layoutlmv3.yaml
├── generate_product_review.yaml
├── graph_parsing.yaml
├── invoices_receipts_ocr_v1.yaml
└── layoutlm_resume_data.yaml
schema/annual_report_to_markdown.yaml: Example schema for converting Annual report to Markdown without using OCR.schema/doclaynet_bench: Example schema to convert a sample of DoclayNet dataset to Markdownschema/extract_info_from_invoice.yaml: Example schema for extracting customer data, invoice data from invoices and synthesizing new data from PDF invoicesschema/funsd_layoutlmv3.yaml: Example schema to process a sample of FunSD LayoutLMv3 datasetschema/generate_product_review.yaml: Schema setup for generating and rating product reviews.schema/graph_parsing.yaml: Example schema for extracting insights from static graphsschema/invoices_receipts_ocr_v1.yaml: Example schema for processing invoices and receipts sampled from huggingface dataset.schema/layoutlm_resume_data.yaml: Example schema for parsing candidates info and experience from resumes sample from huggingface dataset.
exports/doclaynet_bench/result.json: Thisjsonlcontains processed sample of DoclayNet Benchmark dataset extracted using Cyyrus CLI.exports/funsd_layoutlmv3/result.json: Thisjsonlcontains processed sample of FunSD LayoutLMv3 dataset generated using Cyyrus CLI.exports/invoices_receipts_ocr_v1/result.json: Thejsonlcontains information about the invoice items, customer information, and invoice questions and answers extracted from invoices-and-receipts dataset using Cyyrus CLI.exports/layoutlm_resume_data/result.json: Thejsonlcontains information about the candidate's personal details, work experience, education, and skills extracted from layoutlm_resume_dataset using Cyyrus CLI.
We get it. Cyyrus is cool and all. But Writing SCHEMA? Manually? In 2024? Bruh, are you serious?
You know what's fun? Literally anything else. You know what's not? Staring at a CLI, questioning your life choices as you painstakingly create yet another YAML file.
So here's the thing: We made a notebook that does the heavy lifting for you. Why? Because we're not sadists, and you're not a masochist (we hope).
Incase we got your hopes high, spoiler alert, this isn't perfect. But gives you a great start. We have custom components. You'd need to know which buttons to smash, and you're golden. Hell, we even made guides for the button-smashing part.
The notebook is so good, you might actually enjoy creating schemas. (Don't worry, we won't tell anyone.) Go ahead. Click it. But this isn't ideal, we are working on getting a neat UI up. Stay tuned!
We're here to give developers what they really need, not what looks good in a TechCrunch headline. We've been there. We've felt the pain, and yes, we've even built some of those well-intentioned but ultimately inadequate tools ourselves. Now, we're channeling that into building Cyyrus.
The current Cyyrus package is experimental and built over the weekend to understand if terraforming data ops is viable. Does our approach resonate with you? ? Or do you think we're completely off base?
Don't hold back - we love to talk, and more importantly, we love to listen. Lessgo.
Wanna lend us a hand? Great! There are many ways you can make Cyyrus awesome.
- Improving the docs: Got a knack for explaining stuff? This is your time to shine! Everything is written in [Markdown] to facilitate the process of contributing.
- Implementing new tasks: Wanna see a new Task? We’re all ears! We love expanding Cyyrus to cover as many use cases as possible.
- Callout our flaws: Spotted a Bug? Awesome. Make some noise. Help us notice. Tell us all about it! What happened? What did you expect to happen instead?
- Rev us up: Got a Feature Idea? Awesome! Tell us all about it! Paint us a picture, why it’s useful, and how you think it should work.
Head over to contributor's guide to know how to get started. We absolutely love pull requests!
Current tooling around running evaluation, performing finetuning are broken. They are built by optimists, dreamers, and in many cases, brilliant engineers. But they're building tools for a world that doesn't exist - a world where data comes pre-cleaned, perfectly labeled, and ready for AI consumption.
Data is unstructured and messy. These $10/month tools? They're useless for 80% of your data. Sure, the tool costs $10/month. But what about the army of data scientist you need to make your data "tool-ready”.
Cyyrus plans to introduce components which makes existing tools "data-ready". Think react.email for last mile data transformation.
Created by engineers who have an irrational belief that data transformation doesn't need to be a clusterfuck as it is today.
Terraform made infrastructure sexy. They believe a tool like Cyyrus can make data transformation sexy as well. With their half built, barely functional MVP, they're curious if they can make others believe as well.