Bản viết lại dựa trên nguồn: https://blog.cloudflare.com/transformers-on-logpush/
Filter, reshape, and redact logs before export with Transformers | Cloudflare Blog Skip to content All CategoriesAIDevelopersRadarProduct NewsSecurityPolicy & LegalZero TrustSpeed & ReliabilityLife at CloudflarePartners Switch Site LanguageEnglishDeutschEspañolEspañol (Latinoamérica)FrançaisItaliano日本語한국어繁體中文简体中文PortuguêsРусскийBahasa IndonesiaภาษาไทยTiếng ViệtPolskiالعربيةעבריתSvenskaNederlandsTürkçe LoginDashboardContact Sales BlogDevelopers1 TagsShow 1 tagsPost TagsDevelopersAll tagsMatching tagsNo tags found1.1.1.12FAAbuseAccessAccess Control Lists (ACLs)AccessibilityAccount TakeoverAcquisitionsAddressingAdvanced Certificate ManagerAdvanced DDoSAdvertisingAegisAEOAfricaAfroflareAgent CloudAgent Development LifecycleAgent ReadinessAgentsAgents WeekAgents Week 2026AIAI BotsAI GatewayAI SearchAI WAFAI WeekAI-SPMAlertmanagerAlways OnlineAMDAMPAnalyticsAnonymousAnti MalwareAnycastAPIAPI GatewayAPI SecurityAPI ShieldAPJCAppleApplication SecurityApplication ServicesArea 1 SecurityArgo Smart RoutingASCIIAsiaAthenian ProjectAtlassianAttacksAudit LogsAustinAustraliaAuthenticationAuthyAuto RagAutomatic HTTPSAutomatic Platform OptimizationAutomationAutoMinifyAwardsAWSBaiduBandwidth AllianceBandwidth CostsBest PracticesBetaBetter InternetBGPBillingBirthday WeekBlack FridayBlackbirdBot Fight ModeBot ManagementBotnetBotsBPFBrandBrand ProtectionBrazilBrowser InsightsBrowser RenderingBrowser RunBug BountyBugsBYOIPCacheCache PurgeCache ReserveCache RulesCaliforniaCanadaCap'n ProtoCAPTCHACareersCASBCategoriesCDNCDNJSCertificate AuthorityCertificate PinningCertificate TransparencyCertificationCFSSLChallenge PageChatGPTChinaChina NetworkChristmasChromeCIO WeekCISAClaireCLIClickHouseClientlessClientless Web IsolationCloud ConnectorCloud Email SecurityCloudflare AccessCloudflare AppsCloudflare Area 1Cloudflare CallsCloudflare Email ServiceCloudflare for CampaignsCloudflare for SaaSCloudflare for StartupsCloudflare GatewayCloudflare HistoryCloudflare ImagesCloudflare Media PlatformCloudflare MeetupsCloudflare NetworkCloudflare OneCloudflare One ClientCloudflare One User Risk ScoreCloudflare One WeekCloudflare OSCloudflare PagesCloudflare PolishCloudflare QueuesCloudflare RealtimeCloudflare StreamCloudflare TunnelCloudflare TVCloudflare WorkersCloudflare Workers (PT)Cloudflare Workers KVCloudflare Workers KV (ES)Cloudflare Zero TrustCloudforce OneCloudyCode OrangeCoinbaseColombiaCommunityComplianceCompressionConfig RulesConfiguration ManagementCongestion ControlConnectivityConnectivity CloudConsumer ServicesContainersContent Independence DayContent ScanningContextCoreCOVID-19Crawler HintsCrowdStrikeCrypto WeekCryptographyCSAM ReportingCustomer SuccessCustomer ZeroCustomersCVECVE-2023-50387Cyber ReadinessCybersecurityD1DashboardDataData CatalogData CenterData LocalizationData Localization SuiteData LossData Loss PreventionData PlatformData Privacy DayData ProtectionData SovereigntyData Transfer BucketDatabaseDDoSDDoS AlertsDDoS ReportsDebuggingDeep DiveDescalerDesignDeskopeDeveloper DocumentationDeveloper PlatformDeveloper SpotlightDeveloper WeekDevelopersDevelopers StorageDevice SecurityDevOpsDEXDigital Experience MonitoringDigital ForensicsDisruptDistributedDistributed SystemsDistributed WebDiversityDLPDMARCDNSDNS FilteringDNS FloodDNS SecurityDNSSECDogfoodingDoHDomain RankingsDomain Scoped RolesdosdDrupalDue ProcessDurable ExecutionDurable ObjectsEarly HintsEarth DayeBPFEC2eCommerceEdgeEdge ComputingEdge DatabaseEdge RulesEducationEgressElasticElection SecurityElectionsElliptic CurvesEmailEmail RoutingEmail SecurityEmail WorkersEmDashEmissionsEmployee Resource GroupsEncrypted SNIEncryptionEngineeringEnterpriseEntropyEPYCEthereumEuropeEuropean UnionEventsExploitFacebookFancy BearFast FontsFCCFeature FlagsFedRAMPFedRAMP HighFedRAMP ModerateFirefoxFirewallFirmwareFloridaFootballFormal MethodsForresterFortranFoundation DNSFounders' LetterFranceFraudFreeFreedom of SpeechFront EndFull StackFull Stack WeekFunGA WeekGartnerGatebotGDPRGen XGeneral AvailabilityGenerative AIGeo Key ManagerGermanyGitHubGoGoogleGoogle AnalyticsGoogle CloudGoogle WorkspaceGovernment InnovationGrace HopperGrafanaGraphQLGreenGrinchGrowthgRPCGuest PostHackathonHalloweenHardwareHashiCorpHertzbleedHeuristicsHistoryHolidaysHolocaustHong KongHosting ConHostnamesHTTP2HTTP3HTTPSHuman RightsHurricaneHybrid CloudHyperdriveI'm Under Attack ModeIBMICANNiCloud Private RelayIdentityIETFIETF StandardsIL4Image OptimizationImage RecognitionImage ResizingImage StorageImpactImpact WeekIncident ReportIncident ResponseIndiaIndicators of CompromiseIndonesianInferenceInfrastructureInfrastructure as CodeInsightsIntelInterconnectionInternal DNSInternet PerformanceInternet QualityInternet RegulationInternet ShutdownInternet SummitInternet TrafficInternet TrendsInternship ExperienceIntrusion DetectionInvestorsIoCsiOSIoTIPFSIPsecIPv4IPv6IRAPIsraelItalyIWDJAMstackJapanJavaScriptJengoJengo PolicyJoomlaJudeoflareKafkaKernelKey ValueKeyless SSLKeyTrapKillnetKoreaKubernetesLangChainLatencyLatin AmericaLatinflareLavaRandLazarus groupLeaked Credential ChecksLegalLegal Patents SableLGBTQIA+Life at CloudflareLinuxLisbonLive StreamingLlamaLLMLoad BalancingLocalizationLog PushLog4JLog4ShellLoggingLogsLUAMachine LearningMagecartMagic FirewallMagic Network MonitoringMagic TransitMagic WANMagic WAN ConnectorMalicious JavaScriptMalwareManaged ComponentsManaged RulesMarch of CloudflareMASQUEMCPMeerkatMeetUpMerisMessage ProtocolMexicoMicro-frontendsMicrosoftMicrosoft 365Microsoft AzureMiddle EastMigration HubMilestonesMiniflareMirageMiraiMitelMitigationMixed Content ErrorsMLopsMobileMobile SDKModel Context ProtocolMoldovaMonitoringMulti-CloudMulti-UserMySQLNaaSNet NeutralityNetworkNetwork InterconnectNetwork Performance UpdateNetwork ProtectionNetwork ServicesNetworkingNew YearNGINXNinjasNISTNode.jsNorth AmericaNotebooksNotificationsNSEC3OAuthObservabilityOceaniaOCSPOfficesOktaOlympicsOnboardingOpen APIOpen SourceOpenAIOpenBMCOpenDNSOpenSSLOpenTelemetry OptimizationOrigin RulesOutageOxyPacific NorthwestPage RulesPage ShieldParallelsPartnersPartnershipPassword-reusePasswordsPasswords (PT)PatentsPay Per CrawlPAYGOPaymentsPCI CertifiedPeeringPerformancePerformance OptimizationPhishingphpPhythonPingoraPipelinesPlanetScalePlansPlatform EngineeringPlatform WeekPleskPolicy & LegalPoliticsPortugalPost MortemPost-QuantumPostgresPrecursorPrepared StatementsPrismaPrivacyPrivacy PassPrivacy WeekPrivate IPPrivate NetworkProduct DesignProduct NewsProgrammingProgramming (PT)Project Fair ShotProject GalileoProject Honey PotProject PangeaProject SafekeepingProject TurpentinePrometheusProtocolsProudflareProxyingPublic SectorPythonPython WorkersQuantizationQueuesQUICQUICHEQuicksilverR2R2 Super SlurperRadarRadar AlertsRadar APIRadar MapsRailgunRandomnessRansom AttacksRapid ResetRaspberry PiRate LimitingRC4RDDoSReactReading ListReal-timeRecruitingRegional ServicesRegistrarReliabilityRemote Browser IsolationRemote Desktop Protocol Remote WorkReplicationResearchResolverRestreamingRetreatReverse EngineeringREvilRisk ManagementRoad to Zero TrustRocket LoaderRocksDBRoutingRouting SecurityRPCRPKIRRDNSRSARussiaRustRust WorkersSaaSSAAS SecuritySableSaltSamplingSandboxSASESave The WebSDKSearch EngineSecrets StoreSecure Web GatewaySecuritySecurity AnalyticsSecurity CenterSecurity PostureSecurity Posture ManagementSecurity Service EdgeSecurity Weeksecurity.txtSEOServer PushServerlessServerless (PT)Serverless AIServerless WeekServersSIEMSigned Exchanges (SXG)SIMSingaporeSingle Sign On (SSO)Smart PlacementSmart ShieldSnippetsSOC as a ServiceSouth AfricaSouth AmericaSpainspdySpectrumSpeedSpeed & ReliabilitySpeed BrainSpeed WeekSpoofingSportsSQLSRESSESSHSSLStandardsStartup Enterprise PlanStatisticsStopTheHackerStorageSumo LogicSuper BowlSupercloudSupply Chain AttacksSupportSustainabilitySWAGSWGSwiftSwitzerlandSXSWSYNSYN FloodSyriaTCPTeamTeams DashboardTech TalksTechCrunchTechnical WritingTerraformTestimonialsTestingTexasThanksgivingThe Serverlist NewsletterThreat DataThreat FeedsThreat IntelligenceThreat OperationsThreat ReportThreatsTiered CacheTikTokTLSTLS 1.3ToolsTorTracingTrafficTransform RulesTransparencyTrendsTrust & SafetyTTFBTTLTURNTURN ServerTurnstileTypeScriptUDPUkraineUnited KingdomUniversal SSLURL ScannerUSAUser ResearchVDIVectorizeVetflareVideoVisibilityViteVoIPVPCVPNVulnerabilitiesWAFWAF Attack ScoreWAF RulesWaiting RoomWARPWARP ConnectorWASMWeb Application FirewallWeb Asset DiscoveryWeb3WebAssemblyWebinarsWebMCPWebPWebRTCWebSocketsWildebeestWomenflareWordPressWorkers AIWorkers LaunchpadWorkers LogsWorkers ObservabilityWorkers SitesWorkers UnboundWorkers VPCWorkflowsWorld IPv6 DayWranglerx402Year in ReviewZ3ZarazZero Day ThreatsZero TrustZero Trust WeekZone VersioningArtificial IntelligenceWorkersClient-Side SecurityArchitectureMulti-tenant SecuityMulti-tenant SecuritycfBEACONCrawlersDevelopers October 12, 2026Filter, reshape, and redact logs before export with Transformers Rob Taylor and Sahidya Devadoss8 minute readCOPY URL Logpush delivers logs in batches as quickly as possible, enabling Cloudflare to provide information almost in real time. During our recent Birthday Week, we made Logpush available on all self-serve plans and Transformers generally available.Those changes were among eight major updates to Cloudflare Observability and reflect our commitment to make every Cloudflare feature available to everyone. This post takes a deeper look at how Transformers work inside Logpush's export pipeline. Observability teams use Logpush to export Cloudflare logs to security information and event management systems (SIEMs), analytics platforms, and object storage. Before those logs reach their destination, teams may need to remove health checks, omit sensitive fields, or reshape records for dashboards and investigations. Today, this often requires a separate processing service that teams must deploy, scale, monitor, and maintain.For example, a team that collects logs from multiple CDNs can add a static provider field to normalize records before they reach its analytics platform. Teams also pay to export, ingest, and store records they discard as soon as they arrive, so filtering them before delivery can reduce downstream data volume and cost.With Transformers, that processing moves into the Logpush export workflow.Teams define a SQL query to filter records, redact sensitive data, and reshape the output by adding, renaming, removing, or nesting fields. Logpush applies the query before export, so each destination receives data in the form it needs. We made this workflow simple for customers by embedding a SQL transformation engine in Logpush's high-throughput export pipeline while keeping performance and resource use predictable.How Logpush Transformers workA Transformer is a SQL query written for a specific Logpush dataset. The query references fields from that dataset and defines which records and fields Logpush should export. Logpush applies the query to each batch before export. Records that do not match the WHERE clause are removed. For the remaining records, the output of the SELECT statement becomes the exported record. Transformers can change the structure and contents of each record, and the transformed results are exported in NDJSON format.Customers can create, preview, and manage Transformers in the Cloudflare dashboard through Transformer Studio or programmatically through the API. After saving a Transformer, they can attach it to a compatible NDJSON Logpush job that uses the same dataset. Each update creates a new version, so customers can review how a Transformer has changed over time.The following example reshapes a Cloudflare HTTP request log to match a team-defined schema used across CDN providers. It does not convert between serialization formats: both the input and output use NDJSON.{ "ClientIP": "203.0.113.42", "ClientRequestHost": "example.com", "ClientRequestMethod": "POST", "ClientRequestPath": "/api/checkout", "ClientRequestUserAgent": "Mozilla/5.0", "EdgeResponseStatus": 502, "EdgeStartTimestamp": "2026-05-21T15:00:00Z", "RayID": "8e2a1c60ef9e1c9a" } The Transformer renames and nests fields, redacts part of the client IP address, adds provider and error information, removes unused fields, and filters out internal and health-check traffic:SELECT extract(epoch FROM to_timestamp(EdgeStartTimestamp)) AS unix_ts, regexp_replace(ClientIP, '.d+.d+$', '.XXX.XXX') AS redacted_ip, UPPER(ClientRequestMethod) AS method, EdgeResponseStatus AS status, RayID AS ray_id, 'cloudflare' AS provider, named_struct( 'host', ClientRequestHost, 'path', ClientRequestPath ) AS request, EdgeResponseStatus >= 500 AS is_server_error FROM http_requests WHERE ClientRequestHost NOT IN ( 'internal.example.com', 'health.example.com' ) AND ClientRequestPath NOT LIKE '/healthz%' AND ClientRequestPath NOT LIKE '/metrics%'The resulting NDJSON record uses the normalized schema:{ "unix_ts": 1779375600, "redacted_ip": "203.0.XXX.XXX", "method": "POST", "status": 502, "ray_id": "8e2a1c60ef9e1c9a", "provider": "cloudflare", "request": { "host": "example.com", "path": "/api/checkout" }, "is_server_error": true }How Transformers executes SQL with Apache DataFusion and ArrowCloudflare's data pipeline ingests more than one billion events per second. Its Logpush delivery service, which we call Bufstream, delivers more than 110 million Logpush records per second to customer destinations. It runs across hundreds of Kubernetes pods, each claiming a time slice of data from Buftee, Cloudflare's distributed log-based in-house message broker. This model distributes work horizontally across Logpush jobs. Job volume follows a Pareto-like distribution: most jobs operate at lower volumes, while a small number of high-volume jobs process millions of logs each second. Transformers needed to meet the same operational requirements as the rest of Logpush: reliable delivery, high throughput, and predictable resource use without introducing additional failure scenarios into the export hot path.Our initial prototype ran customer-defined WebAssembly Transformers in Cloudflare Workers, which provides globally distributed, isolated compute. However, invoking a Worker would insert a network call into the Logpush export path. That remote dependency could introduce transient timeouts and retries, reducing delivery throughput and increasing delivery latency.We decided to keep Transformer execution in-process within our delivery service to prioritize reliability and delivery throughput. We evaluated several SQL execution engines and selected Apache DataFusion, which already powers Cloudflare Basin.Apache DataFusion is a Rust query engine built on Apache Arrow, with a SQL planner and vectorized, columnar execution engine. In a benchmark, executing a prepared Transformer against a synthetic 5,000-row http_requests batch from a single Go goroutine took about 3 ms, equivalent to roughly 1.6 million rows per second.Attaching a Transformer to a job adds CPU work to its processing path. Bufstream uses per-job transform parallelism so high-volume jobs can execute transform batches concurrently. Embedding DataFusion in Bufstream lets those batches use the service's existing delivery infrastructure, including its scheduling, batching, and destination controls, without introducing a separate transformation tier or network hop.Choosing DataFusion meant connecting Bufstream, which is written in Go, to an engine written in Rust. The next challenge was moving typed batches efficiently across that boundary.The final design of the in-process Transformer pipeline is shown in the diagram below. From records to columnsBufstream receives compressed Logpush batches containing Cap'n Proto records from Buftee. For a job with a Transformer attached, Bufstream decompresses the batch, parses the eligible logs, and builds a typed Apache Arrow RecordBatch before executing the transform.Apache Arrow is a language-independent, columnar in-memory format designed for efficient analytics on modern CPUs and GPUs. It represents flat and nested data as typed, contiguous columns. Arrow IPC gives Bufstream and DataFusion a standard way to exchange batches across the foreign function interface (FFI) boundary between Go and Rust, exposed through a C application binary interface (ABI) and called from Go using cgo, Go's mechanism for calling C code.The Go and Rust runtimes do not share ownership of transform data. Go passes a pointer to its input Arrow IPC buffer to Rust only for the duration of the cgo call, and Rust does not retain that pointer after returning. Rust owns the transformed output buffer; Go copies it into Go-managed memory and then explicitly releases the Rust allocation. This makes ownership and lifetimes explicit across the language boundary, avoiding cross-runtime memory-management bugs.Using Arrow IPC means the Rust DataFusion wrapper can use an Arrow StreamReader to read typed RecordBatch values, rather than parse individual rows from an intermediate format such as JSON and rebuild typed Arrow columns for execution. Field types, nullability, and nested values are represented consistently on both sides of the boundary, so DataFusion receives data in its native columnar model.Crossing the Go/Rust boundary is not free: batches still need to be serialized, decoded, and managed in memory. Arrow IPC keeps that work batch-oriented and avoids per-record parsing and conversion. This makes the boundary a better fit for Logpush's high-throughput delivery pipeline while retaining schema validation against the Logpush dataset definition.Every Arrow batch needs a schema describing its fields and types. Rather than infer that schema from the logs in a particular batch, Bufstream supplies a generated schema from each Logpush dataset's specification. This gives DataFusion a stable view of the available fields and types, enabling query validation. The table named in a Transformer's FROM clause selects which dataset schema the query uses. For example, this Transformer queries the http_requests dataset:SELECT ClientIP, EdgeResponseStatus, RayID FROM http_requests WHERE EdgeResponseStatus >= 500Executing the transformWhen a customer saves a Transformer through the API or Transformer Studio, Cloudflare stores its SQL query in R2. Bufstream retrieves the query through a multi-layer cache and translates it into a reusable DataFusion logical plan that defines the scan, filters, projections, and expressions. Bufstream prepares this plan once per time slice of a job's data and reuses it for every Arrow batch in that slice. For every new Arrow batch, the Rust wrapper binds the batch as the plan’s table input, executes the query, and returns Arrow batches containing the selected records and fields.Transformers are designed to fail closed. If a transform cannot be executed, Bufstream neither advances the job's delivery offset nor sends the untransformed batch. Delivering raw data in place of transformed data could violate the schema, filtering, or redaction assumptions of downstream pipelines, so Bufstream opts to re-attempt failed transforms instead.To prevent a persistently failing job from consuming capacity needed by healthy jobs, Bufstream progressively delays retries and reports errors on the job. Jobs that remain in a failing state for an extended period are automatically disabled. Customers can monitor successful and failed transformations, data volume before and after transformation, and transformation duration in the Transformer Health view of the Health tab in Logpush.Returning to the export pipelineAfter execution, Bufstream converts the Arrow output into NDJSON logs. It batches, gzip-compresses, and uploads those logs to the configured destination. Destinations accept data at different rates and can temporarily slow down. Bufstream adjusts the number of upload goroutines while respecting destination concurrency limits.By embedding SQL execution in Bufstream, Transformers use Logpush’s existing distributed infrastructure, batching, and destination controls. Cloudflare’s data pipeline handles that complexity, while customers create and apply a Transformer with only a few steps.Get startedOpen Transformer Studio from the Logpush page in the Cloudflare dashboard and follow these steps:Choose the Logpush dataset you want to transform.Write your SQL query and preview it against a sample record.Save the Transformer.Attach it to a matching Logpush job.Transformers is available on Free, Pro, Business, and Enterprise plans. See the Transformers documentation for setup details and the Logpush pricing documentation for current pricing information. On this page Discuss OnlineRelated tagsDevelopersFollow on Social MediaCloudflareSubscribe to receive notifications of new postsEmail addressWe’ll never share your email address.SubscribeThanks for subscribing! Check your inbox to confirm. Getting StartedPlansContact salesPartnersFind a partnerStartupsUnder attack?Domain name searchPublic interestProject GalileoAthenian ProjectCloudflare for CampaignsProject FairshotImpact/ESGResourcesApp innovation reportCloudflare RadarCase studiesStatusSupportEventsBlogSolutionsSSE and SASE platformCloudflare AI CloudAI SecurityFrontend Development PlatformMulti-Tenant Platform DevelopmentWeb Security PlatformCompanyAboutCareersInvestorsPressPress kitGlobal networkComplianceCompliance resourcesTrustGDPRResponsible AITransparency reportReport abuseDevelopersDocumentationLearning centerCommunityStart buildingLogin© 2026 Cloudflare, Inc.|Your privacy choicesReport security issues|Privacy Policy|Terms of use|GDPR|Trademark Search is temporarily unavailable.ProductsSolutionsResourcesPricingLogin opens in a new tabDashboard opens in a new tabContact Sales opens in a new tabStart Building opens in a new tab opens in a new tab opens in a new tab opens in a new tabAll CategoriesAIDevelopersRadarProduct NewsSecurityPolicy & LegalZero TrustSpeed & ReliabilityLife at CloudflarePartnersEnglishSwitch Site LanguageEnglishDeutschEspañolEspañol (Latinoamérica)FrançaisItaliano日本語한국어繁體中文简体中文PortuguêsРусскийBahasa IndonesiaภาษาไทยTiếng ViệtPolskiالعربيةעבריתSvenskaNederlandsTürkçeLightDark