Getting Started

Install

pip install ms-rags-all-in-one==1.0.1
ms-rags

Use the PyPI package when you want the released CLI without cloning the repository.

Development install from source

git clone https://github.com/M-SAAD-BIN-MAZHAR/ms-rags-all-in-one-.git
cd ms-rags-all-in-one-
python -m venv .venv

# Windows
.venv\Scripts\activate

# Linux or macOS
source .venv/bin/activate

pip install -e ".[production]"

Use .[production] when you want all providers, vector databases, evaluators, rerankers, and loaders available. Production installs include parser extras for PPTX, Markdown, image OCR, XML, ePub, RTF, Excel, and JSON workflows.

Run The CLI

ms-rags

# Resume a saved session
ms-rags --load session.json

When loading a saved session, MS-RAGS(ALL-IN-ONE) asks for credentials again because secrets are never stored inside the pipeline config JSON.

External Tools You May Need

The Python production install gives you the package integrations, but some document extractors also need command-line tools installed on the machine. MS-RAGS(ALL-IN-ONE) checks your selected loaders and document sources before ingestion, shows what is missing, and asks before continuing.

If you do not want local OCR/PDF tools, choose LlamaParseLoader for complex PDFs and provide LLAMA_CLOUD_API_KEY. Use cloud parsing only when your data policy allows documents to leave the local machine.

Run With Docker

Use Docker when you want a repeatable container environment instead of installing Python packages directly on the host machine.

docker pull saad469/ms-rags-all-in-one
docker run --rm -it \
  --env-file .env \
  -v "${PWD}:/workspace" \
  saad469/ms-rags-all-in-one

Build locally only if you need a custom dependency set.

docker run --rm -it \
  --env-file .env \
  -v "${PWD}/data:/app/data" \
  -v "${PWD}/output:/app/output" \
  saad469/ms-rags-all-in-one

Do not bake API keys into the image. Use an env file, Docker secrets, or your deployment platform's secret manager. Mount local folders for documents, generated output, FAISS indexes, Chroma persistence, graph stores, and agent memory when those features need to survive container restarts.

First Successful Pipeline Tutorial

This path is intentionally conservative. It helps a user get one working RAG system before adding advanced retrieval, reranking, or evaluation.

1. Select one LLM provider. OpenAI, Ollama Cloud, or local Ollama are good first choices.
2. Choose Naive RAG or Advanced RAG. Use Naive if you are learning, Advanced if you already know quality matters.
3. Select one document type first, such as PDF, Markdown, or plain text.
4. Use the recommended default loader. For PDF, start with PyPDFLoader unless tables or layout are complex.
5. Choose Recursive Character chunking with the default size and overlap.
6. Choose OpenAI text-embedding-3-small or a hosted Hugging Face endpoint if you do not want local model downloads.
7. Use ChromaDB for local testing or Qdrant/Pinecone for a production-like managed backend.
8. Approve ingestion only after checking embedding dimension, DB, collection name, and document sources.
9. Use Dense Vector retrieval first. Add Hybrid or reranking only if the retrieved chunks are weak.
10. Ask 5-10 real user questions in the live query loop before generating code.

Query Loop Commands

When query enhancement is enabled, the terminal prints a Query Enhancement Trace with the original query, generated variants, and the exact retrieval query.

The 16-Step Workflow

1. Banner and tracing prompt. Optional OpenTelemetry can be enabled from the terminal.
2. LLM credentials. Select providers and confirm the credential summary.
3. RAG type. Choose exactly one architecture.
4. Document types. Select PDFs, DOCX, CSV, URLs, code, SQL, MongoDB, and more.
5. Loaders. Choose extractors filtered by document type.
6. Chunking strategy. Pick how documents are split.
7. Chunking parameters. Configure size, overlap, tokenizer, separators, or language.
8. Embedding model. Choose local, hosted, or provider-native embeddings.
9. Vector DB and ingestion. Configure storage, test connection, approve ingestion.
10. Query enhancement. Add rewriting, expansion, HyDE, multi-query, step-back, decomposition, or fusion.
11. Retrieval. Choose dense, BM25, TF-IDF, hybrid, MMR, ensemble, parent-child, multi-vector, self-query, or time-weighted.
12. Reranking. Optionally rerank chunks.
13. Compression. Optionally reduce noisy context.
14. System prompt. Confirm grounded answer instructions.
15. Evaluation. Configure quality checks and monitoring.
16. Live query loop. Query, save sessions, then generate standalone code.

User Permission Points

Environment Variables

Only set variables for the providers and databases you actually choose. MS-RAGS(ALL-IN-ONE) asks before using credentials and the generated pipeline reads secrets from the environment instead of storing secret values in code.

OPENAI_API_KEY=sk-...
HUGGINGFACEHUB_API_TOKEN=hf_...
OLLAMA_MODEL_NAME=gpt-oss:120b
OLLAMA_API_KEY=
OLLAMA_BASE_URL=
CHROMA_PERSIST_DIRECTORY=./chroma_db
FAISS_INDEX_PATH=./faiss_indexes/default
QDRANT_URL=http://localhost:6333
PINECONE_API_KEY=...

Generated Output

After the live query loop, MS-RAGS(ALL-IN-ONE) can save a standalone app:

ms_rag_output/
  pipeline.py
  requirements.txt

The generated pipeline supports ingestion, one-shot query, and interactive query mode.

python pipeline.py --ingest --sources ./docs/
python pipeline.py --query "What does this document say?"
python pipeline.py