diff --git a/docs/index.md b/docs/index.md index f24d2fa..30c561f 100644 --- a/docs/index.md +++ b/docs/index.md @@ -16,6 +16,7 @@ pip install vec-inf Otherwise, we recommend using the provided [`vllm.Dockerfile`](https://github.com/VectorInstitute/vector-inference/blob/main/vllm.Dockerfile) and [`sglang.Dockerfile`](https://github.com/VectorInstitute/vector-inference/blob/main/sglang.Dockerfile) to set up your own environment with the package. The built images are available through [Docker Hub](https://hub.docker.com/orgs/vectorinstitute/repositories) If you'd like to use `vec-inf` on your own Slurm cluster, you would need to update the configuration files, there are 3 ways to do it: + * Clone the repository and update the `environment.yaml` and the `models.yaml` file in [`vec_inf/config`](https://github.com/VectorInstitute/vector-inference/blob/main/vec_inf/config), then install from source by running `pip install .`. * The package would try to look for cached configuration files in your environment before using the default configuration. The default cached configuration directory path points to `/model-weights/vec-inf-shared`, you would need to create an `environment.yaml` and a `models.yaml` following the format of these files in [`vec_inf/config`](https://github.com/VectorInstitute/vector-inference/blob/main/vec_inf/config). * [OPTIONAL] The package would also look for an enviroment variable `VEC_INF_CONFIG_DIR`. You can put your `environment.yaml` and `models.yaml` in a directory of your choice and set the enviroment variable `VEC_INF_CONFIG_DIR` to point to that location. diff --git a/docs/user_guide.md b/docs/user_guide.md index ac9a449..66a2761 100644 --- a/docs/user_guide.md +++ b/docs/user_guide.md @@ -37,6 +37,7 @@ You should see an output like the following: ``` **NOTE**: You can set the required fields in the environment configuration (`environment.yaml`), it's a mapping between required arguments and their corresponding environment variables. On the Vector **Killarney** Cluster environment, the required fields are: + * `--account`, `-A`: The Slurm account, this argument can be set to default by setting environment variable `VEC_INF_ACCOUNT`. * `--work-dir`, `-D`: A working directory other than your home directory, this argument can be set to default by seeting environment variable `VEC_INF_WORK_DIR`. @@ -63,12 +64,14 @@ vec-inf launch Meta-Llama-3.1-8B-Instruct --vllm-args '--max-model-len=65536,--c ``` For the full list of inference engine arguments, you can find them here: -* [vLLM](https://docs.vllm.ai/en/stable/serving/engine_args.html). -* [SGLang](https://docs.sglang.io/advanced_features/server_arguments.html) + +* [vLLM: `vllm serve` Arguments](https://docs.vllm.ai/en/stable/serving/engine_args.html) +* [SGLang: Server Arguments](https://docs.sglang.io/advanced_features/server_arguments.html) #### Custom models You can also launch your own custom model as long as the model architecture is supported by the underlying inference engine, and make sure to follow the instructions below: + * Your model weights directory naming convention should follow `$MODEL_FAMILY-$MODEL_VARIANT` ($MODEL_VARIANT is OPTIONAL). * Your model weights directory should contain HuggingFace format weights. * You should specify your model configuration by: