Run prediction and returns predictions and potential metrics. Depending on the dataset and your use case, your test dataset may contain labels. In that case, this method will also return metrics, like in evaluate().

combined (bool, optional, defaults to True) — Creates combined metrics by updating all_results.json with metrics of this call

This also means that if any other tool that is used along the Trainer calls torch.cuda.reset_peak_memory_stats, the gpu peak memory stats could be invalid. And the Trainer will disrupt the normal behavior of any such tools that rely on calling torch.cuda.reset_peak_memory_stats themselves.

Because evaluation calls may happen during train, we can’t handle nested invocations because torch.cuda.max_memory_allocated is a single counter, so if it gets reset by a nested eval call, train’s tracker will report incorrect info. If this pytorch issue gets resolved it will be possible to change this class to be re-entrant. Until then we will only track the outer level of train, evaluate and predict methods. Which means that if eval is called during train, it’s the latter that will account for its memory usage and that of the former.

this tracker doesn’t account for memory allocations outside of Trainer’s __init__, train, evaluate and predict calls.

the very first cuda call typically loads CUDA kernels, which may take from 0.5 to 2GB of GPU memory.

The CPU RAM metric measures RSS (Resident Set Size) includes both the memory which is unique to the process and the memory shared with other processes. It is important to note that it does not include swapped out memory, so the reports could be imprecise.

A class that handles the Trainer control flow. This class is used by the TrainerCallback to activate some switches in the training loop.

on_epoch_begin

trainer = Trainer( model, args, train_dataset=train_dataset, eval_dataset=eval_dataset, callbacks=[MyCallback], # We can either pass the callback class this way or an instance of it (MyCallback()) ) Another way to register a callback is to call trainer.add_callback() as follows: Copied trainer = Trainer(...) trainer.add_callback(MyCallback) # Alternatively, we can pass an instance of the callback class trainer.add_callback(MyCallback())

The argument args, state and control are positionals for all events, all the others are grouped in kwargs. You can unpack the ones you need in the signature of the event using them. As an example, see the code of the simple ~transformer.PrinterCallback. Example: Copied class PrinterCallback(TrainerCallback): def on_log(self, args, state, control, logs=None, **kwargs):

The control object is the only one that can be changed by the callback, in which case the event that changes it should return the modified version.

A class containing the Trainer inner state that will be saved along the model and optimizer when checkpointing and passed to the TrainerCallback.

When using gradient accumulation, one update step may require several forward and backward passes: if you use gradient_accumulation_steps=n, then one update step requires going through n batches.

The main class that implements callbacks is TrainerCallback. It gets the TrainingArguments used to instantiate the Trainer, can access that Trainer’s internal state via TrainerState, and can take some actions on the training loop via TrainerControl.

Callbacks are “read only” pieces of code, apart from the TrainerControl object they return, they cannot change anything in the training loop.

For customizations that require changes in the training loop, you should subclass Trainer and override the methods you need (see trainer for examples).

请问 <pad> 不能通过 attn_mask 来忽略掉，从而相当于模型根本没有见到 <pad> 吗？还是我的理解有什么问题？烦请指教~ 我的理解是：是由于位置编码带来的影响。首先<pad>肯定是可以通过attention_mask将其忽略的，但是如果不考虑序列内部的位置关系的话，其实各个token完全可以乱序放置也不影响对模型的理解与生成。这也是transformer类模型引入位置编码的作用，引入其没有的序列位置信息。 那具体对于 Llama 模型来说，旋转位置编码 (RoPE) 使得在计算注意力的过程中会考虑到不同token之间的相对位置关系，因此如果在 <input> 和 <target> 之间插入过多的 <pad> ，就会破坏这种相对位置关系，并影响模型的生成效果。

MQA是MHA的变体形式，K，V只有一个头，Q是多头，通过共享K得到attention score

bnb_4bit_quant_type (str, {fp4, nf4}, defaults to fp4) — This sets the quantization data type in the bnb.nn.Linear4Bit layers. Options are FP4 and NF4 data types which are specified by fp4 or nf4.

llm_int8_has_fp16_weight (bool, optional, defaults to False) — This flag runs LLM.int8() with 16-bit main weights. This is useful for fine-tuning as the weights do not have to be converted back and forth for the backward pass.

llm_int8_skip_modules (List[str], optional) — An explicit list of the modules that we do not want to convert in 8-bit. This is useful for models such as Jukebox that has several heads in different places and not necessarily at the last position. For example for CausalLM models, the last lm_head is kept in its original dtype.

llm_int8_threshold (float, optional, defaults to 6) — This corresponds to the outlier threshold for outlier detection as described in LLM.int8() : 8-bit Matrix Multiplication for Transformers at Scale paper: https://arxiv.org/abs/2208.07339 Any hidden states value that is above this threshold will be considered an outlier and the operation on those values will be done in fp16. Values are usually normally distributed, that is, most values are in the range [-3.5, 3.5], but there are some exceptional systematic outliers that are very differently distributed for large models. These outliers are often in the interval [-60, -6] or [6, 60]. Int8 quantization works well for values of magnitude ~5, but beyond that, there is a significant performance penalty. A good default threshold is 6, but a lower threshold might be needed for more unstable models (small models, fine-tuning).

llm_int8_enable_fp32_cpu_offload (bool, optional, defaults to False) — This flag is used for advanced use cases and users that are aware of this feature. If you want to split your model in different parts and run some parts in int8 on GPU and some parts in fp32 on CPU, you can use this flag. This is useful for offloading large models such as google/flan-t5-xxl. Note that the int8 operations will not be run on CPU.

device_map (str or Dict[str, Union[int, str, torch.device]] or int or torch.device, optional) — A map that specifies where each submodule should go. It doesn’t need to be refined to each parameter/buffer name, once a given module name is inside, every submodule of it will be sent to the same device. If we only pass the device (e.g., "cpu", "cuda:1", "mps", or a GPU ordinal rank like 1) on which the model will be allocated, the device map will map the entire model to this device. Passing device_map = 0 means put the whole model on GPU 0. To have Accelerate compute the most optimized device_map automatically, set device_map="auto".

LOCAL_RANK - The rank of the worker within a local worker group.

Make sure Python is installed, put jemdoc in your path somewhere, type in your file, and run jemdoc index.jemdoc This will use a default configuration for the html elements, and create an index.html. Even simpler, you can omit the extension, and jemdoc will still process the index.jemdoc file, as in jemdoc index

Note that ZeRO3 is not currently supported with QLoRA but ZeRO3 does support LoRA, which has a reference configuraiton under playground/deepspeed_config_s3.json

# restore the model file "model.h5" from a specific run by user "lavanyashukla"# in project "save_and_restore" from run "10pr4joa"best_model = wandb.restore('model.h5', run_path="lavanyashukla/save_and_restore/10pr4joa")

The result is that the flattened subarrays are sorted in lexicographic order starting with the first element.

Returns: fprndarray of shape (>2,)Increasing false positive rates such that element i is the false positive rate of predictions with score >= thresholds[i]. tprndarray of shape (>2,)Increasing true positive rates such that element i is the true positive rate of predictions with score >= thresholds[i]. thresholdsndarray of shape (n_thresholds,)Decreasing thresholds on the decision function used to compute fpr and tpr. thresholds[0] represents no instances being predicted and is arbitrarily set to np.inf.

fprndarray of shape (>2,)Increasing false positive rates such that element i is the false positive rate of predictions with score >= thresholds[i]. tprndarray of shape (>2,)Increasing true positive rates such that element i is the true positive rate of predictions with score >= thresholds[i].

drop_intermediatebool, default=TrueWhether to drop some suboptimal thresholds which would not appear on a plotted ROC curve. This is useful in order to create lighter ROC curves. New in version 0.17

y_scorearray-like of shape (n_samples,)Target scores, can either be probability estimates of the positive class, confidence values, or non-thresholded measure of decisions (as returned by “decision_function” on some classifiers).

y_truearray-like of shape (n_samples,)True binary labels. If labels are not either {-1, 1} or {0, 1}, then pos_label should be explicitly given.

To group the indices by element, rather than dimension, use argwhere, which returns a row for each non-zero element.

Create a Precision-Recall curve in one line:wandb.log({"pr": wandb.plot.pr_curve(ground_truth, predictions)})You can log this whenever your code has access to:a model's predicted scores (predictions) on a set of examplesthe corresponding ground truth labels (ground_truth) for those examples(optionally) a list of the labels/class names (labels=["cat", "dog", "bird"...] if label index 0 means cat, 1 = dog, 2 = bird, etc.)(optionally) a subset (still in list format) of the labels to visualize in the plot

Note that resuming a run which was executed as part of a Sweep is not supported.

If you set WANDB_RESUME equal to "allow", you can always set WANDB_RUN_ID to a unique string and restarts of the process will be handled automatically. If you set WANDB_RESUME equal to "must", W&B will throw an error if the run to be resumed does not exist yet instead of auto-creating a new run.

wandb.restoreThis will allow you to log new historical values for your metrics to a run starting from where you left off but does not take care of re-establishing the state of your code, you will need to make sure you have written checkpoints that you can load!

We provide a utility to generate run_id: wandb.util.generate_id()

The other form of resume requires you to provide the actual run id: wandb.init(id=run_id)

runs can be resumed by passingresume=True to wandb.init(). This can be thought of as auto-resuming, where we “automatically” pick up from where an aborted run left off.

if you want to be sure that it is resuming, you do wandb.init(id=run_id, resume="must"

Note: This only works if you are running your script in the same directory as the one that failed as the file is stored at: wandb/wandb-resume.json.

quick way to generate a new id: python -c "import wandb; print(wandb.util.generate_id())"

a helper method wandb.util.generate_id() which can be used to generate a random set of characters to append to your experiment name.

unfortunately today you can not re-use deleted run ids in a project.

@zaccharieramzi , could you also add the flag --include-offline when syncing all offline runs? The cmd should be: wandb sync --include-synced --include-offline --sync-all. The reason we need to include this flag is that in offline mode all your runs have a prefix offline (you can check these out in the wandb dir or the zip file you shared above). Therefore, the code at this line suggests that if this flag is provided, all runs in offline-run-* will be included when syncing to W&B.

For performance, we recommend setting bias to None first, and then lora_only, before trying all.

The weight matrix is scaled by lora_alpha/r, and a higher lora_alpha value assigns more weight to the LoRA activations.

The steps are very similar to the ones shown in this quickstart; prepare a PeftConfig for a 🤗 PEFT method, and use the get_peft_model to create a PeftModel from the configuration and base model. Then you can train it however you like!

+ config = PeftConfig.from_pretrained(peft_model_id) model = AutoModelForSeq2SeqLM.from_pretrained(config.base_model_name_or_path) + model = PeftModel.from_pretrained(model, peft_model_id)

To get a sense of the number of trainable parameters in your model, use the print_trainable_parameters method.

inference_mode, whether you’re using the model for inference or not

Run paths include entity, project, and run ID, in the format entity/project/run_id.

when you send your model to prepare, it is wrapped in the DistributedDataParallel class which will reserve memory for the gradients it will need to reduce-average at the end of the backward pass.

Saving the entire 16bit model weights to directly load later on using model.load_state_dict(torch.load(pytorch_model.bin)). For this, either set zero_optimization.stage3_gather_16bit_weights_on_model_save to True in DeepSpeed Config file or set zero3_save_16bit_model to True in DeepSpeed Plugin.

all these functions require ~2x memory (general RAM) of the size of the final checkpoint.

To get 32bit weights, first save the model using model.save_checkpoint(). Below is the snippet from examples/by_feature/deepspeed_with_config_support.py showing this:

Should only be used when wanting to save a checkpoint during training and restoring the state in the same environment.

If a ProjectConfiguration was passed to the Accelerator object with automatic_checkpoint_naming enabled then checkpoints will be saved to self.project_dir/checkpoints. If the number of current saves is greater than total_limit then the oldest save is deleted. Each checkpoint is saved in seperate folders named checkpoint_<iteration>.

the underlying load function, such as optional arguments for DeepSpeed’s load_checkpoint function or a map_location to load the model and optimizer on.

device_map="auto" will be good enough as 🤗 Accelerate will attempt to fill all the space in your GPU(s), then loading them to the CPU, and finally if there is not enough RAM it will be loaded to the disk (the absolute slowest option).

The first step is to init an empty skeleton of the model which won’t take up any RAM using the init_empty_weights() context manager: Copied from accelerate import init_empty_weights with init_empty_weights(): my_model = ModelClass(...) With this my_model currently is “parameterless”, hence leaving the smaller footprint than what one would normally get loading this onto the CPU directly.

Additional key word arguments passed along to the wandb.init method.

Following the recommended settings for Python’s configparser, Flake8 does not support inline comments for any of the keys. So while this is fine:

Not every Flake8 command-line option can be specified in the configuration file. See our list of options to determine which options will be parsed from the configuration files.

we expect you to use INI to configure Flake8 (since each of these files already uses INI as a format). This means that any Flake8 configuration you wish to set needs to be in the flake8 section, which means it needs to start like so: [flake8] Each command-line option that you want to specify in your config file can be named in either of two ways: Using underscores (_) instead of hyphens (-) Simply using hyphens (without the leading hyphens)

Regardless of whether you keep your config in .flake8, setup.cfg, or tox.ini we expect you to use INI to configure Flake8 (since each of these files already uses INI as a format).

New in 3.0.0 The user can specify --append-config <path-to-file> repeatedly to include extra configuration files that should be read and take precedence over user and project files.

New in 3.0.0 The user can specify --config <path-to-file> to so this file is the only configuration file used. This is a change from Flake8 2 where pep8 would simply merge this configuration file into the configuration generated by user and project files (where this takes precedence).

New in 3.0.0 The user can specify --isolated to disable configuration via discovered configuration files.

By default Black looks for pyproject.toml starting from the common base directory of all files and directories passed on the command line. If it’s not there, it looks in parent directories. It stops looking when it finds the file, or a .git directory, or a .hg directory, or the root of the file system, whichever comes first. If you’re formatting standard input, Black will look for configuration starting from the current working directory. You can use a “global” configuration, stored in a specific location in your home directory. This will be used as a fallback configuration, that is, it will be used if and only if Black doesn’t find any configuration as mentioned above. Depending on your operating system, this configuration file should be stored as: Windows: ~\.black Unix-like (Linux, MacOS, etc.): $XDG_CONFIG_HOME/black (~/.config/black if the XDG_CONFIG_HOME environment variable is not set) Note that these are paths to the TOML file itself (meaning that they shouldn’t be named as pyproject.toml), not directories where you store the configuration. Here, ~ refers to the path to your home directory. On Windows, this will be something like C:\\Users\UserName.

run the Output: Focus on Output command in the Command Palette and then select the formatter extension channel

black does not support formatting sections of code.

There are two ways to save a file to associate with a run.Use wandb.save(filename).Put a file in the wandb run directory, and it will get uploaded at the end of the run.

在单个输入token上转发LLM所需的时间，与在K个输入token上批量转发LLM所需的时间相同（K比你想象的要大）。这个不直观的事实是因为采样受到内存的严重限制，大部分「工作」不计算，而是将Transformer的权重从VRAM读取到芯片上缓存中进行处理。

The LoRA layer for embeddings might not work as well on the output projection layer

A temporary fix involves the same solution of freezing the embeddings. This fix is not satisfactory for use cases where new tokens need to be added and corresponding representations tuned. A more general fix would be adding LoRA layers to the embeddings or allowing only the new embeddings to be trained.

If you get an this issue ("illegal memory access") then you should use a newer HF LLaMA conversion or downgrade your PyTorch version.

Currently, using bnb_4bit_compute_type='fp16' can lead to instabilities. For 7B LLaMA, only 80% of finetuning runs complete without error. We have solutions, but they are not integrated yet into bitsandbytes.

Resuming a LoRA training run with the Trainer currently not supported by HF.

4-bit inference is slow. Currently, our 4-bit inference implementation is not yet integrated with the 4-bit matrix multiplication

如何下载指定版本的内容呢？在snaphot_download方法中，提供了allow_regex和ignore_regex两个参数，简单来说前者是对指定的匹配项进行下载，后者是忽略指定的匹配项，下载其余部分。

If you don't want to preface the docker command with sudo, create a Unix group called docker and add users to it. When the Docker daemon starts, it creates a Unix socket accessible by members of the docker group.

The docker group grants root-level privileges to the user. For details on how this impacts security in your system, see Docker Daemon Attack Surface.

By default it's the root user that owns the Unix socket, and other users can only access it using sudo.

.to(accelerator.device)

accelerator.process_index

It is if you want to use more than one GPU, using python script.py will only launch one process, you have to use accelerate launch or python -m torch.distributed.launch to use all your GPUs.

Currently the default verbosity of the library is set to WARNING. To change the level of verbosity, use one of the direct setters.

Dictionary with split names as keys (‘train’, ‘test’ for example), and Dataset objects as values. It also has dataset transform methods like map or filter, to process all the splits at once.

If batched is True and batch_size is n > 1, then the function takes a batch of n examples as input and can return a batch with n examples, or with an arbitrary number of examples. Note that the last batch may have less than n examples. A batch is a dictionary, e.g. a batch of n examples is {"text": ["Hello there !"] * n}.

When downloading a dataset, you should download it first on the main process and then load the cached dataset afterward load_dataset will perform a lock under the hood to stop multiple downloads from happening at once, but if you are downloading something not using this library you should use this method. Copied with accelerator.main_process_first(): datasets = load_dataset("glue", "mrpc") Under the hood this is the same as calling:

Local components are optimized based on an overall feedback signal:SGD optimizes weights in a neural net to reduce its training lossNeural architecture search optimizes architectures and hyperparameters to have low validation lossPolicy gradient optimizes policy neural nets to choose actions that lead to high expected rewards

process-based ML systems have better differential capabilities: They help us apply ML to tasks where we don’t have access to outcomes. These tasks include long-range forecasting, policy decisions, and theoretical research.

Outer alignment asks the question - "What should we aim our model at?" In other words, is the model optimizing for the correct reward such that there are no exploitable loopholes? It is also known as the reward misspecification problem.

Inner alignment asks the question - “Is the model trying to do what humans want it to do?”, or in other words can we robustly aim our AI optimizers at any objective function at all?

AI Boxing is attempts, experiments, or proposals to isolate ("box") a powerful AI (~AGI) where it can't interact with the world at large, save for limited communication with its human liaison. It is often proposed that so long as the AI is physically isolated and restricted, or "boxed", it will be harmless even if it is an unfriendly artificial intelligence (UAI).

Codex 能够创建和理解代码，因此可以使用它来执行任务，例如解释文件中的代码的用途。 实现此目的的一种方法是在以“此函数”或“此应用程序是”开头的函数之后添加注释。Codex 通常会将此理解为说明的开始，并补全文本的其余部分。

测试应用程序通常需要使用示例数据。 由于 Codex 是一种能够理解如何编写和理解自然语言的语言模型，因此可以指示 Codex 创建数据，如虚构名称、产品和其他变量的数组。 例如，此处我们要求 Codex 创建天气温度数组。

在大多数情况下，将 API 温度设置为 0 或接近 0（如 0.1 或 0.2）往往会得到更好的结果。 在 GPT-3 模型中，较高的温度可以提供有用的创意结果和随机结果，Codex 模型则不同，较高温度可能会导致收到十分随机或难以预测的响应。 如果需要 Codex 提供不同的潜在结果，请从 0 开始，然后向上递增 0.1，直到找到合适的变体。

对于某些语言，注释的样式可以提高输出的质量。 例如，使用 Python 时，在某些情况下，使用 doc 字符串（三引号括起来的注释）比使用井号 (#) 得到的结果质量更高。

tiktoken is a fast BPE tokeniser for use with OpenAI's models.

引文越接近它支持的文本，模型预测引文所需的距离就越短，这表明内联引文比内容末尾的引文更适合缓解虚假内容的生成。

有时，系统消息输入“仅写出真实事实”或“不捏造信息”可能不足以缓解问题。 相反，要求模型响应同时包含引文有助于减少错误响应的概率。

使用提示指定输出结构时，可能会对结果的性质和质量产生重大影响。

如果不确定要使用哪种语法，请考虑使用 Markdown 或 XML。 这些模型已通过 XML 和 Markdown 的大量 Web 内容进行了训练，这可能会提供更好的结果。

在此简单示例中，将任务从一个步骤分解为两个步骤的效果并不是非常明显，但当尝试将其应用于包含许多事实性陈述的大段文本时，将任务分解就会产生很大的不同。

这是指在提示的末尾包含几个字词或短语，以获取遵循所需形式的模型响应。 例如，使用 “Here’s a bulleted list of key points:\n- ” 等提示有助于确保输出的格式为项目符号列表。

尽管通常仍建议遵循此方法，但与之前的模型版本（GPT-3 和更早）相比，我们的测试表明，无论是否使用该技术，ChatGPT 和 GPT-4 模型的模型响应都是相同的。 在下面的示例中，我们看到添加了语句“几个消息来源... 爆发”到提示的开头或末尾后，不会导致最终模型响应发生任何变化。

模型可能容易受到近因偏差的影响，在此上下文中，这意味着提示末尾的信息可能比提示开头的信息对输出具有更大的影响。 因此，值得尝试的是，在提示结束时重复指令，并评估对生成的响应的影响。

Recently, there have been clear changes in the open-source policy and regulations of our overall organization's code, data, and models. Despite this, we have still worked hard to obtain opening the weights of the model first, but the data involves stricter auditing and is in review with our legal team . Our researchers have no authority to publicly release them without authorization. Thank you for your understanding.

The following repositories are used in xFormers, either in close to original form or as an inspiration: Sputnik GE-SpMM Triton LucidRain Reformer RevTorch Nystromformer FairScale Pytorch Image Models CUTLASS Flash-Attention

这并不能保证内存的完美利用（浪费现在限制在 4% 以下，仅在最后一个块中），但它明显改善了当今业界广泛使用的提前分配方案的浪费 。

FLAT-Attention 与 FlashAttention 采取不同的路线来解决同一问题。 提出的解决方案有所不同，但关键思想是相同的（tiling 和 scheudling）。下面主要讨论二者不同之处：

torch.cuda.empty_cache()[source] Releases all unoccupied cached memory currently held by the caching allocator so that those can be used in other GPU application and visible in nvidia-smi. Note empty_cache() doesn’t increase the amount of GPU memory available for PyTorch. However, it may help reduce fragmentation of GPU memory in certain cases.

PagedAttention, an attention algorithm inspired by the classic idea of virtual memory and paging in operating systems. Unlike the traditional attention algorithms, PagedAttention allows storing continuous keys and values in non-contiguous memory space. Specifically, PagedAttention partitions the KV cache of each sequence into blocks, each block containing the keys and values for a fixed number of tokens. During the attention computation, the PagedAttention kernel identifies and fetches these blocks efficiently.

Dynamic: Its size depends on the sequence length, which is highly variable and unpredictable. As a result, efficiently managing the KV cache presents a significant challenge. We find that existing systems waste 60% – 80% of memory due to fragmentation and over-reservation.

The KV cache is Large: Takes up to 1.7GB for a single sequence in LLaMA-13B.

past_key_values (tuple(tuple(torch.FloatTensor)), optional, returned when use_cache=True is passed or when config.use_cache=True) — Tuple of tuple(torch.FloatTensor) of length config.n_layers, with each tuple having 2 tensors of shape (batch_size, num_heads, sequence_length, embed_size_per_head)) and 2 additional tensors of shape (batch_size, num_heads, encoder_sequence_length, embed_size_per_head)

TruthfulQA：用于测试模型复制网上常见虚假内容的倾向

HellaSwag：常识推理测试，但对大语言模型来说具有相当的挑战性

AI2 ：针对科学问题的推理测试

斯坦福大学对Alpaca-7B 的全面微调是在8 个 A100 80GB 上进行的，并花费了3个小时

研究团队对模型的微调最初主要针对的是注意力模块，如 v_proj、q_proj、k_proj 和 o_proj。后来，研究人员转向了对gate_proj、down_proj 和 up_proj 模块的微调，与注意力模块相比，除了可训练参数小于总参数的 0.1% 时，微调这些模块模型的性能表现更好。为了保持一致性，研究团队对13B和70B模型统一采用了这一方法，可训练参数分别为0.27%和0.2%。

一个13B的鸭嘴兽模型可以在单个A100 GPU使用25k个问题在5小时内完成训练。

70B模型使用4个A100 80GB进行了22个小时的微调

Open-Platypus由11个开源数据集组成，主要由人为设计的问题组成，只有大约10%的问题由LLM生成，能够以最小的微调时间和成本实现强大的性能。侧重于提高LLM的STEM和逻辑能力。

Contrary to RNNs that have the position of each token embedded within them, transformers are unaware of the position of each token. Therefore, the position IDs (position_ids) are used by the model to identify each token’s position in the list of tokens.

run_on_remote.py is a script that launches any example on remote self-hosted hardware, with automatic hardware and environment setup.

subfolder (str, optional, defaults to "") — In case the relevant files are located inside a subfolder of the model repo on huggingface.co, you can specify the folder name here.

safe_serialization (bool, optional, defaults to False) — Whether to save the model using safetensors or the traditional PyTorch way (that uses pickle).

is_main_process (bool, optional, defaults to True) — Whether the process calling this is the main process or not. Useful when in distributed training like TPUs and need to call this function on all processes. In this case, set is_main_process=True only on the main process to avoid race conditions.

state_dict (nested dictionary of torch.Tensor) — The state dictionary of the model to save. Will default to self.state_dict(), but can be used to only save parts of the model or if special precautions need to be taken when recovering the state dictionary of a model (like when using model parallelism).

A path or url to a PyTorch state_dict save file (e.g, ./pt_model/pytorch_model.bin). In this case, from_pt should be set to True and a configuration object should be provided as config argument.

If the torchscript flag is set in the configuration, can’t handle parameter sharing so we are cloning the weights instead.

Takes care of tying weights embeddings afterwards if the model class has a tie_weights() method.

Returns torch.nn.Embedding Pointer to the input tokens Embeddings Module of the model.

Solutions that failed to reach an answer within 1024 tokens were discarded, resulting in less than 1860 samples on some problems.

For some datasets it can be much faster to yield batches of data rather than examples one by one. You can speed up the dataset generation by yielding Arrow tables directly, instead of examples. This is especially useful if your data comes from Pandas DataFrames for example, since the conversion from Pandas to Arrow is as simple as: Copied import pyarrow as pa pa_table = pa.Table.from_pandas(df)

In the former case, the tokens will NOT be removed from the tokenizer’s full vocabulary - they are only being flagged as non-special tokens.

Tokens are only added if they are not already in the vocabulary (tested by checking if the tokenizer assign the index of the unk_token to them).

Represents a token that can be be added to a Tokenizer. It can have special options that defines the way it should behave.

🤗 Datasets uses Arrow for its local caching system. It allows datasets to be backed by an on-disk cache, which is memory-mapped for fast lookup. This architecture allows for large datasets to be used on machines with relatively small device memory. For example, loading the full English Wikipedia dataset only takes a few MB of RAM

Iterating over Wikipedia on a laptop gives you speeds of 1-3 Gbit/s:

Arrow enables large amounts of data to be processed and moved quickly. It is a specific data format that stores data in a columnar memory layout. This provides several significant advantages: Arrow’s standard format allows zero-copy reads which removes virtually all serialization overhead. Arrow is language-agnostic so it supports different programming languages. Arrow is column-oriented so it is faster at querying and processing slices or columns of data. Arrow allows for copy-free hand-offs to standard machine learning tools such as NumPy, Pandas, PyTorch, and TensorFlow. Arrow supports many, possibly nested, column types.

We have already created Beam pipelines for some of the larger datasets like wikipedia, and wiki40b.

The processing pipeline is executed on a distributed processing backend such as Apache Flink, Apache Spark, or Google Cloud Dataflow.

Some datasets are too large to be processed on a single machine. Instead, you can process them with Apache Beam, a library for parallel data processing.

Other metrics, such as BLEU are harder to interpret: while they also range between 0 and 1, they can vary greatly depending on which parameters are used to generate the scores, especially when different tokenization and normalization techniques are used (see the metric card for more information about BLEU limitations).

These two types of evaluation can use different metrics and measure different aspects of model performance. For example, offline evaluation can compare a model to other models based on their performance on common benchmarks, whereas online evaluation will evaluate aspects such as latency and accuracy of the model based on production data (for example, the number of user queries that it was able to address).

Having an imbalanced dataset can skew the results of your metrics. Imagine a dataset with 99 “non-fraud” cases and 1 “fraud” case. A simple model that always predicts “non-fraud” cases would give yield a 99% accuracy which might sound good at first until you realize that you will never catch a fraud case.

scipy.stats.bootstrap

Without specifying a device, the default for model inference will be the first GPU on the machine if one is available, and else CPU. If you want to use a specific device you can pass device to compute where -1 will use the GPU and a positive integer (starting with 0) will use the associated CUDA device.

Currently supported tasks are: "text-classification": will use the TextClassificationEvaluator. "token-classification": will use the TokenClassificationEvaluator. "question-answering": will use the QuestionAnsweringEvaluator. "image-classification": will use the ImageClassificationEvaluator. "text-generation": will use the TextGenerationEvaluator. "text2text-generation": will use the Text2TextGenerationEvaluator. "summarization": will use the SummarizationEvaluator. "translation": will use the TranslationEvaluator. "automatic-speech-recognition": will use the AutomaticSpeechRecognitionEvaluator.

You can find examples of dataset structures by consulting the “Dataset Preview” function or the dataset card for a given dataset

You can find the right metric for your task by: Looking at the Task pages to see what metrics can be used for evaluating models for a given task. Checking out leaderboards on sites like Papers With Code (you can search by task and by dataset). Reading the metric cards for the relevant metrics and see which ones are a good fit for your use case. For example, see the BLEU metric card or SQuaD metric card. Looking at papers and blog posts published on the topic and see what metrics they report. This can change over time, so try to pick papers from the last couple of years!

perplexity, which can be used for evaluating different kinds of (unsupervised) generative tasks.

There are 3 high-level categories of metrics: Generic metrics, which can be applied to a variety of situations and datasets, such as precision and accuracy. Task-specific metrics, which are limited to a given task, such as Machine Translation (often evaluated using metrics BLEU or ROUGE) or Named Entity Recognition (often evaluated with seqeval). Dataset-specific metrics, which aim to measure model performance on specific benchmarks: for instance, the GLUE benchmark has a dedicated evaluation metric.

we added a CLI that makes creating a new evaluation module much easier: Copied evaluate-cli create "My Metric" --module_type "metric" This will create a new Space on the 🤗 Hub, clone it locally, and populate it with a template.

Evaluation can be run by loading the EvaluationSuite and calling run() method with a model or pipeline. Copied >>> from evaluate import EvaluationSuite >>> suite = EvaluationSuite.load('mathemakitten/sentiment-evaluation-suite') >>> results = suite.run("huggingface/prunebert-base-uncased-6-finepruned-w-distil-mnli")

EvaluationSuite scripts can be defined as follows, and supports Python code for data preprocessing.

It can be useful to evaluate models on a variety of different tasks to understand their downstream performance. The EvaluationSuite enables evaluation of models on a collection of tasks.

from evaluate.visualization import radar_plot

Currently only "text-classification" is supported with more tasks being added in the future.

The evaluator expects a "text" and "label" column for the data input. If your dataset differs you can provide the columns with the keywords input_column="text" and label_column="label"

results = eval.compute(model_or_pipeline=pipe, data=data, metric=metric, ... label_mapping={"NEGATIVE": 0, "POSITIVE": 1}, ... strategy="bootstrap", n_resamples=200)

evaluate.save("./results/"experiment="run 42", **result, **hyperparams)

With the evaluate.push_to_hub() function, you can easily report evaluation results to the model’s repository:

clf_metrics = evaluate.combine(["accuracy", "f1", "precision", "recall"]) The combine function accepts both the list of names of the metrics as well as an instantiated modules. The compute call then computes each metric:

🤗 Evaluate solves this issue by only computing the final metric on the first node. The predictions and references are computed and provided to the metric separately for each node. These are temporarily stored in an Apache Arrow table, avoiding cluttering the GPU or CPU memory. When you are ready to compute() the final metric, the first node is able to access the predictions and references stored on all the other nodes. Once it has gathered all the predictions and references, compute() will perform the final metric evaluation.

Typically, when a metric score is additive (f(AuB) = f(A) + f(B)), you can use distributed reduce operations to gather the scores for each subset of the dataset. But when a metric is non-additive (f(AuB) ≠ f(A) + f(B)), it’s not that simple. For example, you can’t take the sum of the F1 scores of each data subset as your final metric.

When it comes to computing the actual score there are two main ways to do it: All-in-one Incremental In the incremental approach the necessary inputs are added to the module with EvaluationModule.add() or EvaluationModule.add_batch() and the score is calculated at the end with EvaluationModule.compute(). Alternatively, one can pass all the inputs at once to compute().

All evalution modules come with a range of useful attributes that help to use a module stored in a EvaluationModuleInfo object. Attribute Description description A short description of the evaluation module. citation A BibTex string for citation when available. features A Features object defining the input format. inputs_description This is equivalent to the modules docstring. homepage The homepage of the module. license The license of the module. codebase_urls Link to the code behind the module. reference_urls Additional reference URLs.

With list_evaluation_modules() you can check what modules are available on the hub. You can also filter for a specific modules and skip community metrics if you want. You can also see additional information such as likes:

If you want to make sure you are loading the right type of evaluation (especially if there are name clashes) you can explicitly pass the type: Copied >>> word_length = evaluate.load("word_length", module_type="measurement")

If you’d like to monitor your evaluation metrics during fine-tuning, specify the evaluation_strategy parameter in your training arguments to report the evaluation metric at the end of each epoch: Copied >>> from transformers import TrainingArguments, Trainer >>> training_args = TrainingArguments(output_dir="test_trainer", evaluation_strategy="epoch")

知乎插件提供了三种非常便携的图片上传方式，支持上传 .gif, .png, .jpg 格式，且在图片上传的时候自动在当前 Markdown 光标所在行自动生成图片链接，无需创作者手动管理

插件会自动扫描文本第一个一级头标签之前的内容，将第一个发现的图片链接作为背景图片

文章标题无需手动输入，插件会自动检测文本的第一个一级头标签： # 这是一个标题（必须只是一个#） 然后将其作为标题，改行的内容也不会进入到正文中，如果没有检测到，还需用户手动输入。

链接扫描 😊 若你想在特定的问题下回答，或想修改自己的某个原有回答，就将问题/答案链接以 #! https://... 的格式放置于答案的第一行，发布时，插件会自动扫描识别，发布至相应的问题下，或修改原有的答案。 比如，你想在 轻功是否真的存在，其在科学上可以解释吗？ 该问题下回答问题, 只需将 #! https://www.zhihu.com/question/19602618 若是你已经创作过的答案, 则将答案的链接, 形如: #! https://www.zhihu.com/question/355223335/answer/1003461264 的链接复制至文件顶部即可。 若是你已经创作过的文章，则将文章的链接，形如： #! https://zhuanlan.zhihu.com/p/107810342 若插件没有在首行扫描到链接，则会询问创作者接下来的操作，你可以选择发布新文章，或从收藏夹中选取相应问题，发布至相应问题下：

支持mermaid 转化为图片 { "zhihu.enableMermaidToPng": true, // 设置开启才能生效 "zhihu.mermaidTheme": "dark" // 支持设置 mermaid 图片主题，默认值是 "default" }

我们在高分辨质谱仪上对这 12 款湿厕纸进行了检测，均检测出了成分表里没有标识的防腐剂。对羟基苯乙酮、尼泊金乙酯等

湿厕纸中的‘水’在带来更好清洁体验的同时，其包装内的潮湿环境也给病菌、微生物滋生提供了有利条件。”树兰（杭州）医院皮肤科主任瞿镔表示，一包湿厕纸在频繁打开的过程中，每一次都会与空气接触，久而久之，很难保持无菌的状态，从这个角度看，确实会存在一定的健康隐患——可能会导致皮肤过敏，诱发湿疹等疾病，如果本身患有皮肤疾病，还有可能加重病情。

瞿镔称，不论是肛门还是尿道、阴道，都应保持干燥的状态，与肛门相比，尿道、阴道的皮肤更加娇嫩、敏感，由于我国暂时没有湿厕纸的相关执行标准，且湿厕纸很难保持无菌的状态，加之部分湿厕纸还有杀菌剂、防腐剂等成分，并不建议女性将此作为常用品来擦拭阴道、尿道，若私处菌群失衡、长期潮湿，反而会增加患病风险。“如果大家有使用湿厕纸的习惯，可选用单片独立包装的产品，这样相对安全一些，一般遵循‘湿-干’的步骤即可。”

目前，正规的湿厕纸、湿纸巾，在生产过程中一般都遵循质检总局发布的《一次性使用卫生用品卫生标准》。其卫生标准和普通级口罩一致。也就是说，在卫生方面，湿厕纸擦手、擦嘴没什么问题。

FYI, all comparisons between fp32 and fp16 due to my weird feeling ends up to the same result except the normalization weight.

device_map = infer_auto_device_map(model) This will return a dictionary mapping modules or weights to a device. On a machine with one Titan RTX for instance, we get the following: {'model.decoder.embed_tokens': 0,

with init_empty_weights(): model = AutoModelForCausalLM.from_config(config)

To use GPUs, you need to install the NVIDIA Container Toolkit . We also recommend using NVIDIA drivers with CUDA version 11.8 or higher.

Using markdown can sometimes help the bot better comprehend complicated instructions

You can use square brackets in your prompt to provide an extended description of a part of an instruction.

remove_invalid_values (bool, optional, defaults to model.config.remove_invalid_values) — Whether to remove possible nan and inf outputs of the model to prevent the generation method to crash. Note that using remove_invalid_values can slow down generation.

force_words_ids(List[List[int]] or List[List[List[int]]], optional) — List of token ids that must be generated. If given a List[List[int]], this is treated as a simple list of words that must be included, the opposite to bad_words_ids. If given List[List[List[int]]], this triggers a disjunctive constraint, where one can allow different forms of each word.

constraints (List[Constraint], optional) — Custom constraints that can be added to the generation to ensure that the output will contain the use of certain tokens as defined by Constraint objects, in the most sensible way possible.

A ModelOutput (if return_dict_in_generate=True or when config.return_dict_in_generate=True) or a torch.FloatTensor.

Tensor.view(*shape) → Tensor Returns a new tensor with the same data as the self tensor but of a different shape.

scores (tuple(torch.FloatTensor) optional, returned when output_scores=True is passed or when config.output_scores=True) — Processed prediction scores of the language modeling head (scores for each vocabulary token before SoftMax) at each generation step.