Evaluation conditions for AI models on Fabric Badger

1. Repository description and relation with the article

This repository contains the supplementary material and configuration files associated with the article "Towards Low-Barrier Blockchain Deployment: AI-Driven Automation and Explainability". Its primary purpose is to ensure full transparency and academic reproducibility of the artificial intelligence-assisted generation process. It details the exact parameters, templates, and mechanisms used by the Fabric Badger framework to construct prompts, send them to the models, and validate the results.

The files included in this repository to facilitate reproducibility are:

• input_prompt.txt: An exact example of the complete prompt that the model received during testing.
• structure.json: The final template in which the tool understands and stores the network configurations.
• base_structure.json: The standard structure always used as a baseline before asking the model to generate the network.
• ai_structure.json: The expected format for the model's response to ensure correct data extraction.

2. Hardware details

To evaluate the models presented in the article, the inference tests were executed on the following high-performance hardware:

• Server: Dell PowerEdge R760XA
• Processors: 2 processors (48 cores and 96 threads in total)
• GPUs: 4 NVIDIA L40S (with single-precision capabilities)

Additionally, for the consumer-grade hardware tests, the execution platform was characterised by:

• Processor: Intel Core i7-13620H (2.40 GHz)
• GPU: NVIDIA GeForce RTX 2050 (4GB VRAM)

3. Model configuration details

The framework interacts with an Ollama local service via HTTP requests. The tests were conducted using the engine's default options to ensure the results are representative of standard usage.

Exact models evaluated:

• deepseek-R1:14B
• deepseek-R1:8B
• deepseek-R1:7B
• deepseek-R1:1.5B
• llama3:8B
• qwen2.5:7B
• qwen2.5:14B

Inference parameters (Default options):

• Temperature: 0.7 (moderate creativity)
• Top P: 0.9 (nucleus sampling, 90% cumulative probability threshold)
• Top K: 40 (considers the top 40 tokens)
• Num predict: -1 (no limit, generates until context is exhausted or a stop token is found)
• Repeat penalty: 1.1 (repetition penalty)

4. Prompt construction

To ensure the model stays within the required scope and produces valid responses, the system constructs it dynamically by combining several context layers:

1. Initial system prompt: Defines the assistant's role, the boundaries, and the obligation to return a strict JSON containing the structure and explaination fields.
2. Template injection: The ai_structure.json file is loaded as plain text and inserted into the prompt as a reference example, allowing the model to know the exact schema it must return.
3. Current network state injection: The current state of the project (based on base_structure.json for all the tests) is added to the prompt allowing the model to reason about the current network and apply incremental changes without losing context.
4. User request and output specification: The user's direct instruction is added alongside a final block that reminds the model to respond strictly to the request under the agreed format.

5. How to reproduce the test

To replicate the article's evaluation under equivalent conditions, the following requirement need to be met:

1. Local Ollama-compatible environment.
2. Models listed in Section 3.
3. Specified model configuration.
4. Send the exact content of the input_prompt.txt file included in this repository.

Evaluation prompt:

The specific user instruction injected in the final stage of prompt construction for all the article's evaluations was always exactly the following:

"Create a network with 2 peer organisations (teachers and students), each with 2 peers, and 1 orderer organisation with 3 orderers. Use Fabric CA for cryptographic material and a database that allows for advanced querying."

Once the model's response is obtained, the framework executes a validation process removing reasoning tags (<think>...</think>) for models that include them, cleaning possible Markdown formatting, and extracting the JSON. Lastly, it the internal parser validates if the structure adheres to the formats expected in ai_structure.json.