Introduction

Have you ever found yourself limited by the token constraints of an AI model? Especially when you need detailed output, these limits can be quite frustrating. Today, I want to introduce a technique that has significantly enhanced my workflow: multi-prompt segmentation. This method involves instructing the AI to determine if the response should be split into multiple parts to avoid token limits, using a simple token to indicate continuation, and automatically generating the next request until completion. For those interested in seeing a full implementation, you can explore StoryCraftr, an open-source project where I'm applying these techniques as a learning experience.

What Is Multi-Prompt Segmentation?

Multi-prompt segmentation is essentially a method where you tell the AI to determine if the response should be split into multiple parts to avoid token limits. The AI generates each part with a continuation token and waits for a "next" command, allowing the code to request more until the output is complete. This approach allows you to maximize output and ensures that your full idea or request can be processed without losing important details. When dealing with long-form content, like books or research papers, this method makes it possible to generate detailed, contextually rich sections without being cut off midway by token limits.

Advantages of Multi-Prompt Segmentation

1. Increased Coherence: By instructing the AI to output multiple parts, you ensure that the entire content is generated in a logical sequence, improving coherence.

2. Efficiency in Long-Form Content: This method is particularly useful for generating long-form content like chapters or research sections. By allowing the AI to split output into parts, you can create more thorough content without compromising due to token limits.

3. Simplified Implementation: Instead of breaking up prompts manually, this technique uses a continuation mechanism where the AI itself indicates if more output is needed.

Challenges and Considerations

1. Risk of Infinite Loop: Allowing the AI to generate multiple parts can lead to an infinite loop if the continuation condition is not properly controlled. Setting a maximum number of iterations is crucial to prevent this.

2. Post-Processing: To ensure consistency across all parts, a final post-processing step is recommended. Using OpenAI again to refine the combined output helps maintain the overall quality and coherence.

3. Loss of Context: Although the AI is instructed to continue, sometimes there may be slight loss of context between parts. A simple recap mechanism can help maintain continuity.

How to Implement Multi-Prompt Segmentation

Below is a simplified pseudocode for implementing this technique in Python. Note that this is an abstract representation meant to illustrate the concept:

# Step 1: Define the prompt and the maximum iteration limit
full_prompt = """
Write a detailed synopsis for each chapter of my novel. Start with the introduction of characters,
their backgrounds, motivations, and how they evolve through each chapter.
"""

MAX_ITERATIONS = 3  # Limit to prevent infinite loop

def post_process(combined_output):
    post_processing_prompt =  """
Review the following content for logical coherence and structure without retaining context or using memory. Identify any discrepancies or areas that require refinement. Clean tokens or complementary info.

{combined_output}
"""
    response = call_openai_api(post_processing_prompt)

# Step 2: Function to call the AI with multi-prompt segmentation
def generate_output(full_prompt, max_iterations):
    iteration = 0
    complete_output = ""
    continuation_token = "<CONTINUE>"

    # Add instructions to the prompt for segmentation handling
    prompt = f"If the response exceeds token limits, provide the response in parts, using '{continuation_token}' at the end of incomplete parts. Wait for 'next' to continue.\n{full_prompt}"

    while iteration < max_iterations:
        response = call_openai_api(prompt)  # Replace with actual API call
        complete_output += response

        if continuation_token in response:
            prompt = "next"
            iteration += 1
        else:
            break

    return post_process(complete_output)

# Step 3: Generate the output
final_output = generate_output(full_prompt, MAX_ITERATIONS)
print(final_output)

Explanation

• The pseudocode above takes a long prompt and instructs the AI to generate output in multiple parts if needed.

• The continuation_token is used to determine if more output is needed, and the code automatically prompts the AI with "next" to continue.

• A limit on iterations (MAX_ITERATIONS) is set to prevent an infinite loop.

• After all parts are generated, a post-processing step can be applied to refine and ensure consistency.

Example Post-Processing Prompt

After obtaining the segmented response, a post-processing step can ensure coherence:

post_processing_prompt = """
Review the following content for consistency and coherence. Ensure that all parts flow seamlessly together and enhance any areas that lack clarity or depth.

{combined_output}
"""

Optimized Content Check

An additional technique involves asking the AI to verify the coherence of the content without relying on tokens in context or memory. This significantly reduces token usage since the AI is only verifying rather than generating or interpreting in-depth:

verification_prompt = """
Review the following content for logical coherence and structure without retaining context or using memory. Identify any discrepancies or areas that require refinement.

{combined_output}
"""

By using this verification approach, you can achieve consistency checks at a much lower token cost, as the AI is not actively processing the context for continuation but rather examining the content provided at face value.

Practical Example in StoryCraftr

In StoryCraftr, I implemented multi-prompt segmentation for generating book outlines and detailed character summaries. By instructing the AI to continue outputting in parts if necessary, the AI can handle each component thoroughly, ensuring that characters have depth and plot lines remain coherent.

Advantages in Detail

Effective Management of Token Limits

OpenAI models like GPT-3 and GPT-4 have specific token limits (e.g., 4096 tokens for GPT-3, 8192 tokens for GPT-4). When generating complex or long-form outputs like entire book chapters or detailed papers, it’s easy to exceed these token limits, leading to truncated outputs. By dividing output into parts dynamically, this approach sidesteps the constraints imposed by these token limits, ensuring that each portion of the output is complete and coherent before moving to the next. In practice, this addresses a core issue in generating larger content pieces without sacrificing quality due to length.

Model Continuity and Inference Load

The prompt instructions at the beginning explicitly tell the model to continue in subsequent parts if necessary. This allows the model to maintain a semblance of continuity by adhering to logical breaks, often marked by the <CONTINUE> token. Technically, while each subsequent part starts afresh without any memory of the prior context (since each API call is stateless), the prompting and structure mimic an ongoing thought, improving coherence compared to starting entirely new, independent prompts.

Flexible Depth with Post-Processing

Using the post-processing step to refine the final output is efficient in terms of token usage. Instead of asking the model to regenerate a lengthy narrative while keeping track of continuity, the multi-prompt segmentation allows for each part to be generated independently. The post-processing combines these segments while maintaining context, which is ultimately more cost-effective because the final validation does not need to handle all tokens at once.

Disadvantages and Technical Challenges

Stateless Nature of Calls

The approach presumes that each segment retains some level of context between API calls, but in practice, every API call is stateless. The model relies on the instructions embedded in the prompt rather than any true contextual understanding carried over from the previous segment. This results in a disjointed continuity, especially with descriptive details or multi-character dialogue. Unlike a single, extended prompt that leverages the full internal model state, each continuation part can experience subtle shifts in tone, style, or even specific details.

Risk of Contextual Drift

In AI-generated content, there is an inherent risk of "contextual drift" where, in subsequent parts, the AI deviates from the original direction or intended flow. Even though the post-processing step aims to bring coherence, the underlying problem is that each generated segment may interpret the instructions slightly differently. For instance, with a character-driven plot or technical section of a paper, each part might not align perfectly with the intended narrative or argumentative structure. The technical burden then shifts to either the user or a post-processing step to enforce consistent continuity, which may not always be seamless.

Latency and Computational Efficiency

Multiple iterations involve multiple API calls, each taking time for the round trip. The latency accumulates, making this approach less efficient for real-time or near-instantaneous requirements. Additionally, each API call comes with its own computational cost, which could become prohibitive if applied carelessly without controlling the number of iterations. The proposed limit on iterations (MAX_ITERATIONS) is a safeguard against infinite loops. However, tuning this parameter manually based on the content's length or complexity still requires domain expertise. If this limit is too low, the generated content may be insufficient. If too high, it increases unnecessary computational load, adding inefficiency.

Applicability and Practical Performance

In use cases where this approach applies—such as generating story outlines, chapters, academic paper sections, or even extensive technical documentation—the method works effectively. It allows for an exhaustive level of detail that would be unachievable in a single prompt due to token limitations.

Based on practical experience, including my own usage in the StoryCraftr project, the multi-prompt segmentation technique provides a reliable method for navigating through OpenAI's token constraints. The use of continuation tokens and simple "next" instructions mimics a longer session without sacrificing the quality or consistency of the output, albeit at the cost of potential drift or inefficiencies. In cases where the primary concern is depth and comprehensiveness—such as drafting intricate narratives or academic sections—this technique is more than adequate.

However, its applicability is limited when:

• Real-Time Responsiveness is required: The method adds latency due to multiple calls.
• High Continuity is needed without manual review: Since the AI lacks memory between calls, subtle deviations are often inevitable.

In my use of ChatGPT, I've observed that when employing this segmentation approach with proper continuation instructions, the AI reliably provides coherent and logically connected responses across multiple segments. This is particularly true for creative and structured content where prompts can inherently guide the AI to keep a consistent tone.

Does It Truly Make Sense?

Yes, it does—but only in certain contexts. For tasks involving iterative, detailed generation where coherence, depth, and contextual richness are more valuable than real-time interaction or absolute continuity, the approach is highly effective.

By employing a straightforward continuation token mechanism along with a maximum iteration count and subsequent post-processing, you achieve a method of working around token limits without compromising significantly on output quality. That said, this technique shines best when used alongside user oversight, where generated content can be post-processed for consistency—a capability often less critical in scenarios requiring immediate AI feedback.

Conclusion

Multi-prompt segmentation is a powerful method for overcoming token limitations and enhancing the depth of AI-generated content. Although it has some challenges, such as managing context and ensuring segment continuity, the benefits far outweigh these hurdles when generating detailed long-form content. For those interested in diving deeper, StoryCraftr provides a real-world example of these techniques in action. Stay tuned for more experiments and innovations as I continue exploring the intersection of AI and creative writing.