serena.analytics
Source code: serena/analytics.py
-
class TokenCountEstimator[source]
Bases: ABC
-
abstract estimate_token_count(text)[source]
Estimate the number of tokens in the given text.
This is an abstract method that should be implemented by subclasses.
- Parameters:
text (str)
- Return type:
int
-
class TiktokenCountEstimator(model_name='gpt-4o')[source]
Bases: TokenCountEstimator
Approximate token count using tiktoken.
The tokenizer will be downloaded on the first initialization, which may take some time.
- Parameters:
model_name (str) – see tiktoken.model to see available models.
-
estimate_token_count(text)[source]
Estimate the number of tokens in the given text.
This is an abstract method that should be implemented by subclasses.
- Parameters:
text (str)
- Return type:
int
-
class AnthropicTokenCount(model_name='claude-sonnet-4-20250514', api_key=None)[source]
Bases: TokenCountEstimator
The exact count using the Anthropic API.
Counting is free, but has a rate limit and will require an API key,
(typically, set through an env variable).
See https://docs.anthropic.com/en/docs/build-with-claude/token-counting
- Parameters:
model_name (str)
api_key (str | None)
-
estimate_token_count(text)[source]
Estimate the number of tokens in the given text.
This is an abstract method that should be implemented by subclasses.
- Parameters:
text (str)
- Return type:
int
-
class CharCountEstimator(avg_chars_per_token=4)[source]
Bases: TokenCountEstimator
A naive character count estimator that estimates tokens based on character count.
- Parameters:
avg_chars_per_token (int)
-
estimate_token_count(text)[source]
Estimate the number of tokens in the given text.
This is an abstract method that should be implemented by subclasses.
- Parameters:
text (str)
- Return type:
int
-
class RegisteredTokenCountEstimator(
- value,
- names=<not given>,
- *values,
- module=None,
- qualname=None,
- type=None,
- start=1,
- boundary=None,
)[source]
Bases: Enum
-
classmethod get_valid_names()[source]
Get a list of all registered token count estimator names.
- Return type:
list[str]
-
class ToolUsageStats(token_count_estimator=RegisteredTokenCountEstimator.TIKTOKEN_GPT4O)[source]
Bases: object
A class to record and manage tool usage statistics.
- Parameters:
token_count_estimator (RegisteredTokenCountEstimator)
-
property token_estimator_name: str
Get the name of the registered token count estimator used.
-
class Entry(
- *,
- num_times_called: 'int' = 0,
- input_tokens: 'int' = 0,
- output_tokens: 'int' = 0,
)[source]
Bases: object
- Parameters:
num_times_called (int)
input_tokens (int)
output_tokens (int)
-
update_on_call(input_tokens, output_tokens)[source]
Update the entry with the number of tokens used for a single call.
- Parameters:
input_tokens (int)
output_tokens (int)
- Return type:
None
-
get_stats(tool_name)[source]
Get (a copy of) the current usage statistics for a specific tool.
- Parameters:
tool_name (str)
- Return type:
Entry
