Running OpenCode Agentic AI workflows with local LLM models on NCShare with a vLLM Server
This is a guide on setting up OpenCode to run Agentic AI workflows with local LLM models on NCShare. The model is served through a vLLM server hosted on a GPU node and OpenCode connects to that server as a client over its OpenAI-compatible API.
In certain scenarios you may want to move away from cloud-based LLMs to locally hosted ones for your agentic AI workflows as they offer benefits including data privacy, offline functionality, freedom from subscription fees, API rate limits, and avoid censorship.
In this guide, we will launch a vLLM server on a GPU node, and set up OpenCode to communicate with the model hosted locally for agentic AI workflows. Compared to Ollama, vLLM offers higher throughput and lower latency and handles many concurrent requests well, which suits the many rapid tool-calling round trips an agentic coding session generates. For an Ollama-based version of this workflow, see Running OpenCode Agentic AI workflows with local LLM models on NCShare with an Ollama Server.
Prerequisites
The first step is to set up OpenCode on NCShare as instructed in the tutorial, Setting up OpenCode on NCShare.
The next steps involve setting up vLLM and starting up a vLLM server on a GPU node as described in the guide Running Inference on Local LLMs with a vLLM Server. Essentially, launching OpenCode with a local LLM replaces the Running inference on the model section in that guide. Please follow these steps before proceeding to the next section.
Two changes to that setup are needed for agentic work.
Choose a model trained for tool calling. The Qwen/Qwen2-7B-Instruct default in vllm_server.sh is a general chat model and will not drive an agent reliably. With 141 GB of VRAM per H200 there is room for a capable coder model, so set MODEL_NAME in the script to,
If you would rather start small, Qwen/Qwen2.5-Coder-7B-Instruct also works well; pair it with --tool-call-parser hermes instead of the qwen3_coder parser used below.
Enable tool calling on the server. vLLM does not emit tool calls unless asked to; without it the model's tool-call syntax comes back as ordinary message text and the agent loop never advances. Both --enable-auto-tool-choice and a matching --tool-call-parser are required. vllm_server.sh forwards any extra arguments to vllm serve, so no edit to the launch command is needed,
Once the server is up you will see an output similar to the one below, which tells you the URL it is serving at, the model alias, and the API key,
vLLM is serving at: http://compute-gpu-03:8000
Model: Qwen/Qwen3-Coder-30B-A3B-Instruct-FP8
Model Alias: local-vllm
Tensor parallel: 1x GPU
Extra vLLM args: --enable-auto-tool-choice --tool-call-parser qwen3_coder
Export on client shell:
export VLLM_HOST=http://compute-gpu-03:8000
export VLLM_API_KEY=your-secret-key
Stop server:
kill 567059 && pkill -f VLLM::
Connecting OpenCode to the vLLM server
With the vLLM server running, export the connection details on the client shell where you will run OpenCode, copying them from the server startup output,
Replace compute-gpu-03 and the port with the host and port printed by vllm_server.sh when you started the server. The client does not need a GPU, since the inference engine is doing the heavy work on the GPUs of the server node.
We then set up OpenCode to connect to it to use the local LLM model.
Copy the following opencode.json configuration file to ~/.config/opencode/opencode.json.
{
"$schema": "https://opencode.ai/config.json",
"default_agent": "plan",
"permission": {
"edit": "ask",
"bash": "allow",
"webfetch": "allow"
},
"provider": {
"vllm": {
"npm": "@ai-sdk/openai-compatible",
"name": "vLLM (local)",
"options": {
"baseURL": "{env:VLLM_HOST}/v1",
"apiKey": "{env:VLLM_API_KEY}"
},
"models": {
"local-vllm": {}
}
}
}
}
The vllm provider section here enables OpenCode to reach the model through the server's OpenAI-compatible API. The model key local-vllm must match the SERVED_MODEL_NAME alias set in vllm_server.sh. If you serve a different model or change that alias, update this key to match. The default_agent and permission entries are general OpenCode settings and can be adjusted to taste. The permission block above lets the agent run shell commands and fetch web pages without prompting, but asks before editing files.
Reading baseURL and apiKey from the environment keeps the secret key out of the configuration file and lets you move between server nodes without editing it. Do not hard-code the key into opencode.json.
Important!
If you will be using opencode to perform computationally intensive tasks, first request an interactive session on NCShare and run it on that.
Next, start OpenCode with the command,
and run the /models command there to select a model. You will see the local LLM model served through vLLM under the category, "vLLM (local)".
Now you can run your agentic AI workflows without having to worry about data privacy or exhausting your cloud AI tokens. An example agentic workflow to perform an Equation of State (EOS) analysis can be found in the tutorial, Performing Equation of State Analysis using Agentic AI Skills.
Using other agentic AI tools
OpenCode is not the only option here. The vLLM server exposes a standard OpenAI-compatible API, so any coding agent that can be pointed at a custom OpenAI-style endpoint can share the same server.
Codex CLI is configured the same way through ~/.codex/config.toml,
model = "local-vllm"
model_provider = "vllm"
[model_providers.vllm]
name = "vLLM (local)"
base_url = "http://compute-gpu-03:8000/v1"
env_key = "VLLM_API_KEY"
wire_api = "chat"
As before, model must match the SERVED_MODEL_NAME alias set in vllm_server.sh. Unlike opencode.json, Codex CLI does not expand environment variables inside base_url, so you will need to write out the server node's address (the VLLM_HOST value from the startup output) in full, e.g. http://compute-gpu-03:8000/v1. The env_key entry names the environment variable holding the API key, which is the VLLM_API_KEY you exported from the server startup output; Codex CLI expects it to be non-empty, so set it to a placeholder value if you removed the API_KEY from the server script. Setting wire_api = "chat" tells Codex CLI to use the Chat Completions API rather than the Responses API.
Once you are done with your session, stop the server we started on the GPU node with the kill command provided in the server startup output. E.g.,