Feat/add chunk size control arg (#56)
This pull request addresses issue #38 **Changes included:** - Added a `max_tokens` parameter to the typer cli for the `generate` command, allowing user to set a default number of tokens per chunk. - Introduced a default variable `MAX_TOKEN_PER_CHUNK` to centralize the default value. - Fixed a tokenization issue by inserting a comma, allowing the model to correctly read each chunk as intended rather than treating one chunk as a single sentence. **Future improvement suggestion :** - Consider enhancing the `split_into_best_sentences` function using regex to split text more accurately without modifying it, which would improve clarity and maintainability. --------- Co-authored-by: Václav Volhejn <8401624+vvolhejn@users.noreply.github.com>
This commit is contained in:
co-authored by
Václav Volhejn
parent
36f21a0a7f
commit
4ec89acf95
@@ -1,4 +1,4 @@
|
||||
"""Various utilities for audio convertion (pcm format, sample rate and channels),
|
||||
"""Various utilities for audio conversion (pcm format, sample rate and channels),
|
||||
and volume normalization."""
|
||||
|
||||
import torch
|
||||
|
||||
Reference in New Issue
Block a user