index : renesas/drm.git	android live master
	libdrm, cloned from git://anongit.freedesktop.org/mesa/drm	Laurent Pinchart

summaryrefslogtreecommitdiff

log msg author committer range

path: root/bsd-core

Age	Commit message (Collapse)	Author
2005-08-05	Rename the driver hooks in the DRM to something a little more	Eric Anholt
	understandable: preinit -> load postinit -> (removed) presetup -> firstopen postsetup -> (removed) open_helper -> open prerelease -> preclose free_filp_priv -> postclose pretakedown -> lastclose postcleanup -> unload release -> reclaim_buffers_locked version -> (removed) postinit and version were replaced with generic code in the Linux DRM (drivers now set their version numbers and description in the driver structure, like on BSD). postsetup wasn't used at all. Fixes the savage hooks for initializing and tearing down mappings at the right times. Testing involved at least starting X, running glxgears, killing glxgears, exiting X, and repeating. Tested on: FreeBSD (g200, g400, r200, r128) Linux (r200, savage4)
2005-08-04	Fix the MGA driver on BSD by passing in the proper chipset flags to the	Eric Anholt
	driver's preinit routine, and by using DRM_COPY_TO_USER_IOCTL when copying out to an ioctl's data pointer. Pulled from the latest version of my drm-hook-rename.diff and only compile-tested after that.
2005-07-27	Correct a couple of descriptions of files in comments (were just	Eric Anholt
	copy'n'pasted). Submitted by: jkim
2005-07-27	Link in the savage files from shared-core.	Eric Anholt
	Submitted by: jkim
2005-07-20	Add latest r300 support from r300.sf.net CVS. Patch submitted by volodya,	Eric Anholt
	with BSD fix from jkim and the r300_reg.h license from Nicolai Haehnle. Big thanks to everyone involved!
2005-06-29	Get the BSD DRM compiling again after MGA and mapping changes. Note that	Eric Anholt
	i915 has been disabled for the moment, rather than working around linux-specific code in the shared dir.
2005-06-14	Adds support for PCI cards to MGA DRM	Ian Romanick
	This patch adds serveral new ioctls and a new query to get_param query to support PCI MGA cards. Two ioctls were added to implement interrupt based waiting. With this change, the client-side driver no longer needs to map the primary DMA region or the MMIO region. Previously, end-of-frame waiting was done by busy waiting in the client-side driver until one of the MMIO registers (the current DMA pointer) matched a pointer to the end of primary DMA space. By using interrupts, the busy waiting and the extra mappings are removed. A third ioctl was added to bootstrap DMA. This ioctl, which is used by the X-server, moves a *LOT* of code from the X-server into the kernel. This allows the kernel to do whatever needs to be done to setup DMA buffers. The entire process and the locations of the buffers are hidden from user-mode. Additionally, a get_param query was added to differentiate between G4x0 cards and G550 cards. A gap was left in the numbering sequence so that, if needed, G450 cards could be distinguished from G400 cards. According to Ville Syrjälä, the G4x0 cards and the G550 cards handle anisotropic filtering differently. This seems the most compatible way to let the client-side driver know which card it's own. Doing this very small change now eliminates the need to bump the DRM minor version twice. http://marc.theaimsgroup.com/?l=dri-devel&m=106625815319773&w=2 A number of ioctl handlers in linux-core were also modified so that they could be called in-kernel. In these cases, the in-kernel callable version kept the existing name (e.g., drm_agp_acquire) and the ioctl handler added _ioctl to the name (e.g., drm_agp_acquire_ioctl). This patch also replaces the drm_agp_do_release function with drm_agp_release. drm_agp_release (drm_core_agp_release in the previous patch) is very similar to drm_agp_do_release, and I saw no reason to have both. This commit *breaks the build* on BSD. Eric said that he would make the required updates to the BSD side soon. Xorg bug: 3259 Reviewed by: Eric Anholt
2005-06-09	Completely re-initialize DMA settings	Ian Romanick
	There were two problems. First, the 'warp' and 'primary' pointers weren't cleared, so mga_do_cleanup_dma, which gets called multiple times, would try to ioremapfree them multiple times. This resulted in the new error messages to syslog. The second problem was the, since the dev_private structure isn't reallocated and cleaned out in mga_do_init_dma, when the server is reloaded idle-waits would wait for impossible values. I have given this patch some more riggorous testing. This includes: - Load module, start server, run GL app, stop server, unload module. - Load module, start server, run GL app, stop server, unload module, reload module, restart server, run GL app. - Load module, start server, run GL app, stop server, restart server, run GL app, stop server, unload module. In all three cases, everything worked as expected. Please let me know if there are any further regressions with this patch. Xorg bug: 3408 Reported by: Chris Rankin
2005-06-06	Add a few more bits of Tonnerre's NetBSD port (Still need to deal with the	Eric Anholt
	device attachment).
2005-05-27	Modify drm_driver::device_is_agp to return a tri-state value to indicate	Ian Romanick
	that a device absolutely is, absolutely is not, or may or may not be AGP. Modify the i915 DRM to use this to force all i9x5 devices to be "AGP" (even the PCI-e devices). Reported by: Lukas Hejtmanek
2005-05-21	Change the MGA initialization and cleanup a bit. The dev_private structure	Ian Romanick
	is now allocated (and partially filled in) by the new mga_driver_preinit function. This allows the driver to detect the type of card (i.e., G200 class vs. G400 class) on its own. The chipset value passed to mga_dma_init is now ignored. This same technique is used by the radeon DRM. As a result of this, mga_driver_pretakedown was converted to mga_driver_postcleanup. This routine gets called in some other places than might be expected, and it sets the dev_private pointer to NULL. That little gem took over an hour to track down. :(
2005-05-16	Added device_is_agp callback to drm_driver. This function is called by the	Ian Romanick
	platform-specific drm_device_is_agp function. Added implementation of this function the the Linux-specific portion of the MGA driver to detect PCI G450 cards. Added code to the Linux-specific portion of the generic DRM layer to not initialize AGP infrastructure if the card is not AGP (this matches what already existed in BSD). Bumped the driver date and the driver patch-level for MGA. This mostly fixes bugzilla #3248. The BSD side still needs an implementation of mga_driver_device_is_agp.
2005-04-26	Convert NOMAN to the new preferred spelling NO_MAN to quiet warnings.	Eric Anholt
2005-04-26	Convert BSD code to mostly use bus_dma, the dma abstraction for dealing	Eric Anholt
	with IOMMUs and such. There is one usage of the forbidden vtophys() left in drm_scatter.c which will be fixed up soon. This required a KPI change for drm_pci_alloc/free() to return/use a drm_dma_handle_t that keeps track of os-specific bits, rather than just passing around the vaddr/busaddr/size. Submitted by: Tonnerre Lombard (partially) Tested on: FreeBSD: Rage128 AGP/PCI Linux: Savage4 AGP/PCI
2005-04-24	Fix a panic on startup with non-initmapping drivers by assigning the	Eric Anholt
	drm_ioremap return value to the map handle again. Submitted by: Tor Egge, tegge at freebsd dot org
2005-04-16	Use /*- to begin license blocks in BSD code to reduce diffs against FreeBSD	Eric Anholt
	CVS.
2005-04-13	Fix build on FreeBSD-current, thanks to jhb@.	Eric Anholt
2005-02-19	Merge r1.26 from FreeBSD: Now that mem(4) is a kernel module, we need to	Eric Anholt
	depend on it.
2005-02-14	Use fuword32 for DRM_GET_USER_UNCHECKED when available. May help on 64-bit	Eric Anholt
	platforms. Submitted by: Jung-uk Kim, jkim at niksun dot com
2005-02-14	Use the proper API to get PCI vendor/device number for a dev.	Eric Anholt
	PR: ports/76879 Submitted by: Alex, lesha at intercaf dot ru.
2005-02-13	Fix bad copy'n'pastage of copyrights -- don't disclaim anything for VA	Eric Anholt
	Linux or PI in my copyrights when I should be doing it for myself.
2005-02-13	Add the first bits necessary for a port of savage to FreeBSD. More to	Eric Anholt
	follow later.
2005-02-08	Close a race which could allow for privilege escalation by users with DRI	Eric Anholt
	privileges on Radeon hardware. Essentially, a malicious program could submit a packet containing an offset (possibly in main memory) to be rendered from/to, while a separate thread switched that offset in userspace rapidly between a valid value and an invalid one. radeon_check_and_fixup_offset() would pull the offset in from user space, check it, and spit it back out to user space to be copied in later by the emit code. It would sometimes catch the bad value, but sometimes the malicious program could modify it after the check and get an invalid offset rendered from/to. Fix this by allocating a temporary buffer and copying the data in at once. While here, make the cliprects stuff not do the VERIFYAREA_READ and COPY_FROM_USER_UNCHECKED gymnastics, avoiding a lock order reversal on FreeBSD. Performance impact is negligible -- no difference on r200 to ~1% improvement on rv200 in quake3 tests (P4 1Ghz, demofour at 1024x768, n=4 or 5).
2005-02-07	Restore a debugging message to DRM_DEBUG instead of DRM_ERROR.	Eric Anholt
2005-02-07	Remove some annoying trailing whitespace.	Eric Anholt
2005-02-05	- Implement drm_initmap, and extend it with the resource number to help	Eric Anholt
	FreeBSD. Add drm_get_resource_{start|len} so linux-specific stuff doesn't need to be in shared code. - Fix mach64 build by using __DECONST to work around passing a const pointer to useracc, which is unfortunately not marked const. - Get rid of a lot of maplist code by not having dev->maplist be a pointer, and by sticking the link entries directly in drm_local_map_t rather than having a separate structure for the linked list. - Factor out map uninit and removal into its own routine, rather than duplicating in both drm_takedown() and drm_rmmap(). - Hook up more driver functions, and correct FreeBSD-specific bits of radeon_cp.c, making radeon work. - Baby steps towards using bus_space as we should.
2005-01-17	Add detection of whether the device is AGP by walking the capabilities	Eric Anholt
	list.
2004-12-15	Use SYSCTL_ADD_OID macro instead of calling function directly.	Eric Anholt
	Submitted by: reffie@FreeBSD.org
2004-11-07	Refine the locking of the DRM. Most significant is covering the driver	Eric Anholt
	ioctls with dev_lock, which is a major step toward being able to remove Giant. Covers some new pieces (dev->unique*) in the core, and avoids one call down into system internals with the drm lock held, which is usually bad (FreeBSD LOR #23, #27).
2004-11-07	Don't link in files which no longer exist.	Eric Anholt
2004-11-07	Now that the memory debug code is gone, and all 3 BSDs have M_ZERO, stop	Eric Anholt
	using drm_alloc/drm_free in the core and instead use plain malloc/free.
2004-11-06	Add the drm Makefile and update .cvsignores.	Eric Anholt
2004-11-06	Convert more drivers for bsd-core, moving the ioctl definitions to shared	Eric Anholt
	code. Remove the "drv" from sisdrv, as it's unnecessary. Use the drm_pci functions in i915 instead of per-os implementations of the same. Avoid whitespace within fields in drm_pciids.txt (one of the r300 definitions), since it breaks the bsd pciids script. Tested on sis, mga, r128. i915 needs more work.
2004-11-06	Remove some core stuff that ended up being unnecessary.	Eric Anholt
2004-11-06	Get r128 basically working: Hook up the driver's dma ioctl, use the proper	Eric Anholt
	offset into the driver ioctl array, and don't make the ctx bitmap conditional.
2004-11-06	Move the lock/unlock ioctls to a more logical place, in drm_lock.c.	Eric Anholt
2004-11-06	Connect up r128_ioctls in driver config.	Eric Anholt
2004-11-06	Hook the debug output up to a sysctl, so you can choose to enable at	Eric Anholt
	runtime.
2004-11-06	Add file missed in last commit: Commit first pieces of port to OpenBSD,	Eric Anholt
	done by Martin Lexa (martin at martinlexa dot cz). Now that we've got porting for all three major BSDs (and the fourth being very similar to FreeBSD), move the mostly-duplication drm_os_* files into drmP.h. Remove some cruft from linux heritage and from pieces of the DRM that have since been removed. Note that things are still not quite working for even FreeBSD, but these are first steps at cleanup, and just a WIP checkpoint.
2004-11-06	Commit first pieces of port to OpenBSD, done by Martin Lexa (martin at	Eric Anholt
	martinlexa dot cz). Now that we've got porting for all three major BSDs (and the fourth being very similar to FreeBSD), move the mostly-duplication drm_os_* files into drmP.h. Remove some cruft from linux heritage and from pieces of the DRM that have since been removed. Note that things are still not quite working for even FreeBSD, but these are first steps at cleanup, and just a WIP checkpoint.
2004-11-06	Remove the vestiges of the memory-debug code.	Eric Anholt
2004-11-06	Commit WIP of BSD conversion to core model. Compiles for r128, radeon, but	Eric Anholt
	doesn't run yet. Moves the ioctl definitions for these two drivers back to the shared code -- they aren't OS-specific.
2004-08-24	This patch adds three new ioctl's to the VIA Unichrome/Pro DRM driver:	Erdi Chen
	DRM_IOCTL_VIA_DMA_INIT DRM_IOCTL_VIA_CMDBUFFER DRM_IOCTL_VIA_FLUSH The first ioctl sets up an area in AGP memory that will be used as the ring buffer. The second ioctl copies a command buffer from user space memory to the ring buffer. The third ioctl waits for engine idle until it returns. The motivation for this patch is to avoid the wait for engine idle call before each buffer flush in the current DRI driver. With this patch, the DRI driver can continue to flush its buffer as long as there is free space in the ring buffer. This patch adds an additional copy operation on the command buffer. This buffer copying is necessary to support multiple DRI clients rendering simultaneously. Otherwise, more CPU time will be spent in the busy loop waiting for engine idle between DRI context switch. Even in the single client case, the tradeoff is reasonable in comparision to the kernel call to check for free buffer space for the client to render directly to the ring buffer.
2004-08-17	Merged drmfntbl-0-0-1	Dave Airlie
2004-08-14	Remove unused pcigart/sg header stuff from i915 driver.	Eric Anholt
2004-08-14	Add a "dev" argument to DRIVER_CTX_[CD]TOR. This will be used in an	Eric Anholt
	upcoming commit for the SiS driver.
2004-08-14	Hopefully proper fix for corrupted driver name in memcontrol list.	Eric Anholt
	Reported by: Jung-uk Kim <jkim@niksun.com>
2004-08-13	Enable MTRR usage on AMD64, and use DELAY() instead of rolling our own	Eric Anholt
	udelay code. Submitted by: Jung-uk Kim <jkim@niksun.com>
2004-08-03	bring over fix from i865-agp branch, it now probes the driver, X hangs	Dave Airlie
	box..
2004-07-29	add read/write 16	Dave Airlie

• [prev]
• [next]

generated by cgit v1.2.3 (git 2.25.1) at 2025-08-03 23:20:05 +0000

/**************************************************************************
 *
 * Copyright © 2007 Red Hat Inc.
 * Copyright © 2007-2012 Intel Corporation
 * Copyright 2006 Tungsten Graphics, Inc., Bismarck, ND., USA
 * All Rights Reserved.
 *
 * Permission is hereby granted, free of charge, to any person obtaining a
 * copy of this software and associated documentation files (the
 * "Software"), to deal in the Software without restriction, including
 * without limitation the rights to use, copy, modify, merge, publish,
 * distribute, sub license, and/or sell copies of the Software, and to
 * permit persons to whom the Software is furnished to do so, subject to
 * the following conditions:
 *
 * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
 * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
 * FITNESS FOR A PARTICULAR PURPOSE AND NON-INFRINGEMENT. IN NO EVENT SHALL
 * THE COPYRIGHT HOLDERS, AUTHORS AND/OR ITS SUPPLIERS BE LIABLE FOR ANY CLAIM,
 * DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR
 * OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE
 * USE OR OTHER DEALINGS IN THE SOFTWARE.
 *
 * The above copyright notice and this permission notice (including the
 * next paragraph) shall be included in all copies or substantial portions
 * of the Software.
 *
 *
 **************************************************************************/
/*
 * Authors: Thomas Hellström <thomas-at-tungstengraphics-dot-com>
 *          Keith Whitwell <keithw-at-tungstengraphics-dot-com>
 *	    Eric Anholt <eric@anholt.net>
 *	    Dave Airlie <airlied@linux.ie>
 */

#ifdef HAVE_CONFIG_H
#include "config.h"
#endif

#include <xf86drm.h>
#include <xf86atomic.h>
#include <fcntl.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>
#include <assert.h>
#include <pthread.h>
#include <sys/ioctl.h>
#include <sys/mman.h>
#include <sys/stat.h>
#include <sys/types.h>
#include <stdbool.h>

#include "errno.h"
#ifndef ETIME
#define ETIME ETIMEDOUT
#endif
#include "libdrm_lists.h"
#include "intel_bufmgr.h"
#include "intel_bufmgr_priv.h"
#include "intel_chipset.h"
#include "intel_aub.h"
#include "string.h"

#include "i915_drm.h"

#ifdef HAVE_VALGRIND
#include <valgrind.h>
#include <memcheck.h>
#define VG(x) x
#else
#define VG(x)
#endif

#define VG_CLEAR(s) VG(memset(&s, 0, sizeof(s)))

#define DBG(...) do {					\
	if (bufmgr_gem->bufmgr.debug)			\
		fprintf(stderr, __VA_ARGS__);		\
} while (0)

#define ARRAY_SIZE(x) (sizeof(x) / sizeof((x)[0]))

typedef struct _drm_intel_bo_gem drm_intel_bo_gem;

struct drm_intel_gem_bo_bucket {
	drmMMListHead head;
	unsigned long size;
};

typedef struct _drm_intel_bufmgr_gem {
	drm_intel_bufmgr bufmgr;

	int fd;

	int max_relocs;

	pthread_mutex_t lock;

	struct drm_i915_gem_exec_object *exec_objects;
	struct drm_i915_gem_exec_object2 *exec2_objects;
	drm_intel_bo **exec_bos;
	int exec_size;
	int exec_count;

	/** Array of lists of cached gem objects of power-of-two sizes */
	struct drm_intel_gem_bo_bucket cache_bucket[14 * 4];
	int num_buckets;
	time_t time;

	drmMMListHead named;
	drmMMListHead vma_cache;
	int vma_count, vma_open, vma_max;

	uint64_t gtt_size;
	int available_fences;
	int pci_device;
	int gen;
	unsigned int has_bsd : 1;
	unsigned int has_blt : 1;
	unsigned int has_relaxed_fencing : 1;
	unsigned int has_llc : 1;
	unsigned int has_wait_timeout : 1;
	unsigned int bo_reuse : 1;
	unsigned int no_exec : 1;
	unsigned int has_vebox : 1;
	bool fenced_relocs;

	char *aub_filename;
	FILE *aub_file;
	uint32_t aub_offset;
} drm_intel_bufmgr_gem;

#define DRM_INTEL_RELOC_FENCE (1<<0)

typedef struct _drm_intel_reloc_target_info {
	drm_intel_bo *bo;
	int flags;
} drm_intel_reloc_target;

struct _drm_intel_bo_gem {
	drm_intel_bo bo;

	atomic_t refcount;
	uint32_t gem_handle;
	const char *name;

	/**
	 * Kenel-assigned global name for this object
	 */
	unsigned int global_name;
	drmMMListHead name_list;

	/**
	 * Index of the buffer within the validation list while preparing a
	 * batchbuffer execution.
	 */
	int validate_index;

	/**
	 * Current tiling mode
	 */
	uint32_t tiling_mode;
	uint32_t swizzle_mode;
	unsigned long stride;

	time_t free_time;

	/** Array passed to the DRM containing relocation information. */
	struct drm_i915_gem_relocation_entry *relocs;
	/**
	 * Array of info structs corresponding to relocs[i].target_handle etc
	 */
	drm_intel_reloc_target *reloc_target_info;
	/** Number of entries in relocs */
	int reloc_count;
	/** Mapped address for the buffer, saved across map/unmap cycles */
	void *mem_virtual;
	/** GTT virtual address for the buffer, saved across map/unmap cycles */
	void *gtt_virtual;
	int map_count;
	drmMMListHead vma_list;

	/** BO cache list */
	drmMMListHead head;

	/**
	 * Boolean of whether this BO and its children have been included in
	 * the current drm_intel_bufmgr_check_aperture_space() total.
	 */
	bool included_in_check_aperture;

	/**
	 * Boolean of whether this buffer has been used as a relocation
	 * target and had its size accounted for, and thus can't have any
	 * further relocations added to it.
	 */
	bool used_as_reloc_target;

	/**
	 * Boolean of whether we have encountered an error whilst building the relocation tree.
	 */
	bool has_error;

	/**
	 * Boolean of whether this buffer can be re-used
	 */
	bool reusable;

	/**
	 * Size in bytes of this buffer and its relocation descendents.
	 *
	 * Used to avoid costly tree walking in
	 * drm_intel_bufmgr_check_aperture in the common case.
	 */
	int reloc_tree_size;

	/**
	 * Number of potential fence registers required by this buffer and its
	 * relocations.
	 */
	int reloc_tree_fences;

	/** Flags that we may need to do the SW_FINSIH ioctl on unmap. */
	bool mapped_cpu_write;

	uint32_t aub_offset;

	drm_intel_aub_annotation *aub_annotations;
	unsigned aub_annotation_count;
};

static unsigned int
drm_intel_gem_estimate_batch_space(drm_intel_bo ** bo_array, int count);

static unsigned int
drm_intel_gem_compute_batch_space(drm_intel_bo ** bo_array, int count);

static int
drm_intel_gem_bo_get_tiling(drm_intel_bo *bo, uint32_t * tiling_mode,
			    uint32_t * swizzle_mode);

static int
drm_intel_gem_bo_set_tiling_internal(drm_intel_bo *bo,
				     uint32_t tiling_mode,
				     uint32_t stride);

static void drm_intel_gem_bo_unreference_locked_timed(drm_intel_bo *bo,
						      time_t time);

static void drm_intel_gem_bo_unreference(drm_intel_bo *bo);

static void drm_intel_gem_bo_free(drm_intel_bo *bo);

static unsigned long
drm_intel_gem_bo_tile_size(drm_intel_bufmgr_gem *bufmgr_gem, unsigned long size,
			   uint32_t *tiling_mode)
{
	unsigned long min_size, max_size;
	unsigned long i;

	if (*tiling_mode == I915_TILING_NONE)
		return size;

	/* 965+ just need multiples of page size for tiling */
	if (bufmgr_gem->gen >= 4)
		return ROUND_UP_TO(size, 4096);

	/* Older chips need powers of two, of at least 512k or 1M */
	if (bufmgr_gem->gen == 3) {
		min_size = 1024*1024;
		max_size = 128*1024*1024;
	} else {
		min_size = 512*1024;
		max_size = 64*1024*1024;
	}

	if (size > max_size) {
		*tiling_mode = I915_TILING_NONE;
		return size;
	}

	/* Do we need to allocate every page for the fence? */
	if (bufmgr_gem->has_relaxed_fencing)
		return ROUND_UP_TO(size, 4096);

	for (i = min_size; i < size; i <<= 1)
		;

	return i;
}

/*
 * Round a given pitch up to the minimum required for X tiling on a
 * given chip.  We use 512 as the minimum to allow for a later tiling
 * change.
 */
static unsigned long
drm_intel_gem_bo_tile_pitch(drm_intel_bufmgr_gem *bufmgr_gem,
			    unsigned long pitch, uint32_t *tiling_mode)
{
	unsigned long tile_width;
	unsigned long i;

	/* If untiled, then just align it so that we can do rendering
	 * to it with the 3D engine.
	 */
	if (*tiling_mode == I915_TILING_NONE)
		return ALIGN(pitch, 64);

	if (*tiling_mode == I915_TILING_X
			|| (IS_915(bufmgr_gem->pci_device)
			    && *tiling_mode == I915_TILING_Y))
		tile_width = 512;
	else
		tile_width = 128;

	/* 965 is flexible */
	if (bufmgr_gem->gen >= 4)
		return ROUND_UP_TO(pitch, tile_width);

	/* The older hardware has a maximum pitch of 8192 with tiled
	 * surfaces, so fallback to untiled if it's too large.
	 */
	if (pitch > 8192) {
		*tiling_mode = I915_TILING_NONE;
		return ALIGN(pitch, 64);
	}

	/* Pre-965 needs power of two tile width */
	for (i = tile_width; i < pitch; i <<= 1)
		;

	return i;
}

static struct drm_intel_gem_bo_bucket *
drm_intel_gem_bo_bucket_for_size(drm_intel_bufmgr_gem *bufmgr_gem,
				 unsigned long size)
{
	int i;

	for (i = 0; i < bufmgr_gem->num_buckets; i++) {
		struct drm_intel_gem_bo_bucket *bucket =
		    &bufmgr_gem->cache_bucket[i];
		if (bucket->size >= size) {
			return bucket;
		}
	}

	return NULL;
}

static void
drm_intel_gem_dump_validation_list(drm_intel_bufmgr_gem *bufmgr_gem)
{
	int i, j;

	for (i = 0; i < bufmgr_gem->exec_count; i++) {
		drm_intel_bo *bo = bufmgr_gem->exec_bos[i];
		drm_intel_bo_gem *bo_gem = (drm_intel_bo_gem *) bo;

		if (bo_gem->relocs == NULL) {
			DBG("%2d: %d (%s)\n", i, bo_gem->gem_handle,
			    bo_gem->name);
			continue;
		}

		for (j = 0; j < bo_gem->reloc_count; j++) {
			drm_intel_bo *target_bo = bo_gem->reloc_target_info[j].bo;
			drm_intel_bo_gem *target_gem =
			    (drm_intel_bo_gem *) target_bo;

			DBG("%2d: %d (%s)@0x%08llx -> "
			    "%d (%s)@0x%08lx + 0x%08x\n",
			    i,
			    bo_gem->gem_handle, bo_gem->name,
			    (unsigned long long)bo_gem->relocs[j].offset,
			    target_gem->gem_handle,
			    target_gem->name,
			    target_bo->offset,
			    bo_gem->relocs[j].delta);
		}
	}
}

static inline void
drm_intel_gem_bo_reference(drm_intel_bo *bo)
{
	drm_intel_bo_gem *bo_gem = (drm_intel_bo_gem *) bo;

	atomic_inc(&bo_gem->refcount);
}

/**
 * Adds the given buffer to the list of buffers to be validated (moved into the
 * appropriate memory type) with the next batch submission.
 *
 * If a buffer is validated multiple times in a batch submission, it ends up
 * with the intersection of the memory type flags and the union of the
 * access flags.
 */
static void
drm_intel_add_validate_buffer(drm_intel_bo *bo)
{
	drm_intel_bufmgr_gem *bufmgr_gem = (drm_intel_bufmgr_gem *) bo->bufmgr;
	drm_intel_bo_gem *bo_gem = (drm_intel_bo_gem *) bo;
	int index;

	if (bo_gem->validate_index != -1)
		return;

	/* Extend the array of validation entries as necessary. */
	if (bufmgr_gem->exec_count == bufmgr_gem->exec_size) {
		int new_size = bufmgr_gem->exec_size * 2;

		if (new_size == 0)
			new_size = 5;

		bufmgr_gem->exec_objects =
		    realloc(bufmgr_gem->exec_objects,
			    sizeof(*bufmgr_gem->exec_objects) * new_size);
		bufmgr_gem->exec_bos =
		    realloc(bufmgr_gem->exec_bos,
			    sizeof(*bufmgr_gem->exec_bos) * new_size);
		bufmgr_gem->exec_size = new_size;
	}

	index = bufmgr_gem->exec_count;
	bo_gem->validate_index = index;
	/* Fill in array entry */
	bufmgr_gem->exec_objects[index].handle = bo_gem->gem_handle;
	bufmgr_gem->exec_objects[index].relocation_count = bo_gem->reloc_count;
	bufmgr_gem->exec_objects[index].relocs_ptr = (uintptr_t) bo_gem->relocs;
	bufmgr_gem->exec_objects[index].alignment = 0;
	bufmgr_gem->exec_objects[index].offset = 0;
	bufmgr_gem->exec_bos[index] = bo;
	bufmgr_gem->exec_count++;
}

static void
drm_intel_add_validate_buffer2(drm_intel_bo *bo, int need_fence)
{
	drm_intel_bufmgr_gem *bufmgr_gem = (drm_intel_bufmgr_gem *)bo->bufmgr;
	drm_intel_bo_gem *bo_gem = (drm_intel_bo_gem *)bo;
	int index;

	if (bo_gem->validate_index != -1) {
		if (need_fence)
			bufmgr_gem->exec2_objects[bo_gem->validate_index].flags |=
				EXEC_OBJECT_NEEDS_FENCE;
		return;
	}

	/* Extend the array of validation entries as necessary. */
	if (bufmgr_gem->exec_count == bufmgr_gem->exec_size) {
		int new_size = bufmgr_gem->exec_size * 2;

		if (new_size == 0)
			new_size = 5;

		bufmgr_gem->exec2_objects =
			realloc(bufmgr_gem->exec2_objects,
				sizeof(*bufmgr_gem->exec2_objects) * new_size);
		bufmgr_gem->exec_bos =
			realloc(bufmgr_gem->exec_bos,
				sizeof(*bufmgr_gem->exec_bos) * new_size);
		bufmgr_gem->exec_size = new_size;
	}

	index = bufmgr_gem->exec_count;
	bo_gem->validate_index = index;
	/* Fill in array entry */
	bufmgr_gem->exec2_objects[index].handle = bo_gem->gem_handle;
	bufmgr_gem->exec2_objects[index].relocation_count = bo_gem->reloc_count;
	bufmgr_gem->exec2_objects[index].relocs_ptr = (uintptr_t)bo_gem->relocs;
	bufmgr_gem->exec2_objects[index].alignment = 0;
	bufmgr_gem->exec2_objects[index].offset = 0;
	bufmgr_gem->exec_bos[index] = bo;
	bufmgr_gem->exec2_objects[index].flags = 0;
	bufmgr_gem->exec2_objects[index].rsvd1 = 0;
	bufmgr_gem->exec2_objects[index].rsvd2 = 0;
	if (need_fence) {
		bufmgr_gem->exec2_objects[index].flags |=
			EXEC_OBJECT_NEEDS_FENCE;
	}
	bufmgr_gem->exec_count++;
}

#define RELOC_BUF_SIZE(x) ((I915_RELOC_HEADER + x * I915_RELOC0_STRIDE) * \
	sizeof(uint32_t))

static void
drm_intel_bo_gem_set_in_aperture_size(drm_intel_bufmgr_gem *bufmgr_gem,
				      drm_intel_bo_gem *bo_gem)
{
	int size;

	assert(!bo_gem->used_as_reloc_target);

	/* The older chipsets are far-less flexible in terms of tiling,
	 * and require tiled buffer to be size aligned in the aperture.
	 * This means that in the worst possible case we will need a hole
	 * twice as large as the object in order for it to fit into the
	 * aperture. Optimal packing is for wimps.
	 */
	size = bo_gem->bo.size;
	if (bufmgr_gem->gen < 4 && bo_gem->tiling_mode != I915_TILING_NONE) {
		int min_size;

		if (bufmgr_gem->has_relaxed_fencing) {
			if (bufmgr_gem->gen == 3)
				min_size = 1024*1024;
			else
				min_size = 512*1024;

			while (min_size < size)
				min_size *= 2;
		} else
			min_size = size;

		/* Account for worst-case alignment. */
		size = 2 * min_size;
	}

	bo_gem->reloc_tree_size = size;
}

static int
drm_intel_setup_reloc_list(drm_intel_bo *bo)
{
	drm_intel_bo_gem *bo_gem = (drm_intel_bo_gem *) bo;
	drm_intel_bufmgr_gem *bufmgr_gem = (drm_intel_bufmgr_gem *) bo->bufmgr;
	unsigned int max_relocs = bufmgr_gem->max_relocs;

	if (bo->size / 4 < max_relocs)
		max_relocs = bo->size / 4;

	bo_gem->relocs = malloc(max_relocs *
				sizeof(struct drm_i915_gem_relocation_entry));
	bo_gem->reloc_target_info = malloc(max_relocs *
					   sizeof(drm_intel_reloc_target));
	if (bo_gem->relocs == NULL || bo_gem->reloc_target_info == NULL) {
		bo_gem->has_error = true;

		free (bo_gem->relocs);
		bo_gem->relocs = NULL;

		free (bo_gem->reloc_target_info);
		bo_gem->reloc_target_info = NULL;

		return 1;
	}

	return 0;
}

static int
drm_intel_gem_bo_busy(drm_intel_bo *bo)
{
	drm_intel_bufmgr_gem *bufmgr_gem = (drm_intel_bufmgr_gem *) bo->bufmgr;
	drm_intel_bo_gem *bo_gem = (drm_intel_bo_gem *) bo;
	struct drm_i915_gem_busy busy;
	int ret;

	VG_CLEAR(busy);
	busy.handle = bo_gem->gem_handle;

	ret = drmIoctl(bufmgr_gem->fd, DRM_IOCTL_I915_GEM_BUSY, &busy);

	return (ret == 0 && busy.busy);
}

static int
drm_intel_gem_bo_madvise_internal(drm_intel_bufmgr_gem *bufmgr_gem,
				  drm_intel_bo_gem *bo_gem, int state)
{
	struct drm_i915_gem_madvise madv;

	VG_CLEAR(madv);
	madv.handle = bo_gem->gem_handle;
	madv.madv = state;
	madv.retained = 1;
	drmIoctl(bufmgr_gem->fd, DRM_IOCTL_I915_GEM_MADVISE, &madv);

	return madv.retained;
}

static int
drm_intel_gem_bo_madvise(drm_intel_bo *bo, int madv)
{
	return drm_intel_gem_bo_madvise_internal
		((drm_intel_bufmgr_gem *) bo->bufmgr,
		 (drm_intel_bo_gem *) bo,
		 madv);
}

/* drop the oldest entries that have been purged by the kernel */
static void
drm_intel_gem_bo_cache_purge_bucket(drm_intel_bufmgr_gem *bufmgr_gem,
				    struct drm_intel_gem_bo_bucket *bucket)
{
	while (!DRMLISTEMPTY(&bucket->head)) {
		drm_intel_bo_gem *bo_gem;

		bo_gem = DRMLISTENTRY(drm_intel_bo_gem,
				      bucket->head.next, head);
		if (drm_intel_gem_bo_madvise_internal
		    (bufmgr_gem, bo_gem, I915_MADV_DONTNEED))
			break;

		DRMLISTDEL(&bo_gem->head);
		drm_intel_gem_bo_free(&bo_gem->bo);
	}
}

static drm_intel_bo *
drm_intel_gem_bo_alloc_internal(drm_intel_bufmgr *bufmgr,
				const char *name,
				unsigned long size,
				unsigned long flags,
				uint32_t tiling_mode,
				unsigned long stride)
{
	drm_intel_bufmgr_gem *bufmgr_gem = (drm_intel_bufmgr_gem *) bufmgr;
	drm_intel_bo_gem *bo_gem;
	unsigned int page_size = getpagesize();
	int ret;
	struct drm_intel_gem_bo_bucket *bucket;
	bool alloc_from_cache;
	unsigned long bo_size;
	bool for_render = false;

	if (flags & BO_ALLOC_FOR_RENDER)
		for_render = true;

	/* Round the allocated size up to a power of two number of pages. */
	bucket = drm_intel_gem_bo_bucket_for_size(bufmgr_gem, size);

	/* If we don't have caching at this size, don't actually round the
	 * allocation up.
	 */
	if (bucket == NULL) {
		bo_size = size;
		if (bo_size < page_size)
			bo_size = page_size;
	} else {
		bo_size = bucket->size;
	}

	pthread_mutex_lock(&bufmgr_gem->lock);
	/* Get a buffer out of the cache if available */
retry:
	alloc_from_cache = false;
	if (bucket != NULL && !DRMLISTEMPTY(&bucket->head)) {
		if (for_render) {
			/* Allocate new render-target BOs from the tail (MRU)
			 * of the list, as it will likely be hot in the GPU
			 * cache and in the aperture for us.
			 */
			bo_gem = DRMLISTENTRY(drm_intel_bo_gem,
					      bucket->head.prev, head);
			DRMLISTDEL(&bo_gem->head);
			alloc_from_cache = true;
		} else {
			/* For non-render-target BOs (where we're probably
			 * going to map it first thing in order to fill it
			 * with data), check if the last BO in the cache is
			 * unbusy, and only reuse in that case. Otherwise,
			 * allocating a new buffer is probably faster than
			 * waiting for the GPU to finish.
			 */
			bo_gem = DRMLISTENTRY(drm_intel_bo_gem,
					      bucket->head.next, head);
			if (!drm_intel_gem_bo_busy(&bo_gem->bo)) {
				alloc_from_cache = true;
				DRMLISTDEL(&bo_gem->head);
			}
		}

		if (alloc_from_cache) {
			if (!drm_intel_gem_bo_madvise_internal
			    (bufmgr_gem, bo_gem, I915_MADV_WILLNEED)) {
				drm_intel_gem_bo_free(&bo_gem->bo);
				drm_intel_gem_bo_cache_purge_bucket(bufmgr_gem,
								    bucket);
				goto retry;
			}

			if (drm_intel_gem_bo_set_tiling_internal(&bo_gem->bo,
								 tiling_mode,
								 stride)) {
				drm_intel_gem_bo_free(&bo_gem->bo);
				goto retry;
			}
		}
	}
	pthread_mutex_unlock(&bufmgr_gem->lock);

	if (!alloc_from_cache) {
		struct drm_i915_gem_create create;

		bo_gem = calloc(1, sizeof(*bo_gem));
		if (!bo_gem)
			return NULL;

		bo_gem->bo.size = bo_size;

		VG_CLEAR(create);
		create.size = bo_size;

		ret = drmIoctl(bufmgr_gem->fd,
			       DRM_IOCTL_I915_GEM_CREATE,
			       &create);
		bo_gem->gem_handle = create.handle;
		bo_gem->bo.handle = bo_gem->gem_handle;
		if (ret != 0) {
			free(bo_gem);
			return NULL;
		}
		bo_gem->bo.bufmgr = bufmgr;

		bo_gem->tiling_mode = I915_TILING_NONE;
		bo_gem->swizzle_mode = I915_BIT_6_SWIZZLE_NONE;
		bo_gem->stride = 0;

		if (drm_intel_gem_bo_set_tiling_internal(&bo_gem->bo,
							 tiling_mode,
							 stride)) {
		    drm_intel_gem_bo_free(&bo_gem->bo);
		    return NULL;
		}

		DRMINITLISTHEAD(&bo_gem->name_list);
		DRMINITLISTHEAD(&bo_gem->vma_list);
	}

	bo_gem->name = name;
	atomic_set(&bo_gem->refcount, 1);
	bo_gem->validate_index = -1;
	bo_gem->reloc_tree_fences = 0;
	bo_gem->used_as_reloc_target = false;
	bo_gem->has_error = false;
	bo_gem->reusable = true;
	bo_gem->aub_annotations = NULL;
	bo_gem->aub_annotation_count = 0;

	drm_intel_bo_gem_set_in_aperture_size(bufmgr_gem, bo_gem);

	DBG("bo_create: buf %d (%s) %ldb\n",
	    bo_gem->gem_handle, bo_gem->name, size);

	return &bo_gem->bo;
}

static drm_intel_bo *
drm_intel_gem_bo_alloc_for_render(drm_intel_bufmgr *bufmgr,
				  const char *name,
				  unsigned long size,
				  unsigned int alignment)
{
	return drm_intel_gem_bo_alloc_internal(bufmgr, name, size,
					       BO_ALLOC_FOR_RENDER,
					       I915_TILING_NONE, 0);
}

static drm_intel_bo *
drm_intel_gem_bo_alloc(drm_intel_bufmgr *bufmgr,
		       const char *name,
		       unsigned long size,
		       unsigned int alignment)
{
	return drm_intel_gem_bo_alloc_internal(bufmgr, name, size, 0,
					       I915_TILING_NONE, 0);
}

static drm_intel_bo *
drm_intel_gem_bo_alloc_tiled(drm_intel_bufmgr *bufmgr, const char *name,
			     int x, int y, int cpp, uint32_t *tiling_mode,
			     unsigned long *pitch, unsigned long flags)
{
	drm_intel_bufmgr_gem *bufmgr_gem = (drm_intel_bufmgr_gem *)bufmgr;
	unsigned long size, stride;
	uint32_t tiling;

	do {
		unsigned long aligned_y, height_alignment;

		tiling = *tiling_mode;

		/* If we're tiled, our allocations are in 8 or 32-row blocks,
		 * so failure to align our height means that we won't allocate
		 * enough pages.
		 *
		 * If we're untiled, we still have to align to 2 rows high
		 * because the data port accesses 2x2 blocks even if the
		 * bottom row isn't to be rendered, so failure to align means
		 * we could walk off the end of the GTT and fault.  This is
		 * documented on 965, and may be the case on older chipsets
		 * too so we try to be careful.
		 */
		aligned_y = y;
		height_alignment = 2;

		if ((bufmgr_gem->gen == 2) && tiling != I915_TILING_NONE)
			height_alignment = 16;
		else if (tiling == I915_TILING_X
			|| (IS_915(bufmgr_gem->pci_device)
			    && tiling == I915_TILING_Y))
			height_alignment = 8;
		else if (tiling == I915_TILING_Y)
			height_alignment = 32;
		aligned_y = ALIGN(y, height_alignment);

		stride = x * cpp;
		stride = drm_intel_gem_bo_tile_pitch(bufmgr_gem, stride, tiling_mode);
		size = stride * aligned_y;
		size = drm_intel_gem_bo_tile_size(bufmgr_gem, size, tiling_mode);
	} while (*tiling_mode != tiling);
	*pitch = stride;

	if (tiling == I915_TILING_NONE)
		stride = 0;

	return drm_intel_gem_bo_alloc_internal(bufmgr, name, size, flags,
					       tiling, stride);
}

/**
 * Returns a drm_intel_bo wrapping the given buffer object handle.
 *
 * This can be used when one application needs to pass a buffer object
 * to another.
 */
drm_intel_bo *
drm_intel_bo_gem_create_from_name(drm_intel_bufmgr *bufmgr,
				  const char *name,
				  unsigned int handle)
{
	drm_intel_bufmgr_gem *bufmgr_gem = (drm_intel_bufmgr_gem *) bufmgr;
	drm_intel_bo_gem *bo_gem;
	int ret;
	struct drm_gem_open open_arg;
	struct drm_i915_gem_get_tiling get_tiling;
	drmMMListHead *list;

	/* At the moment most applications only have a few named bo.
	 * For instance, in a DRI client only the render buffers passed
	 * between X and the client are named. And since X returns the
	 * alternating names for the front/back buffer a linear search
	 * provides a sufficiently fast match.
	 */
	for (list = bufmgr_gem->named.next;
	     list != &bufmgr_gem->named;
	     list = list->next) {
		bo_gem = DRMLISTENTRY(drm_intel_bo_gem, list, name_list);
		if (bo_gem->global_name == handle) {
			drm_intel_gem_bo_reference(&bo_gem->bo);
			return &bo_gem->bo;
		}
	}

	bo_gem = calloc(1, sizeof(*bo_gem));
	if (!bo_gem)
		return NULL;

	VG_CLEAR(open_arg);
	open_arg.name = handle;
	ret = drmIoctl(bufmgr_gem->fd,
		       DRM_IOCTL_GEM_OPEN,
		       &open_arg);
	if (ret != 0) {
		DBG("Couldn't reference %s handle 0x%08x: %s\n",
		    name, handle, strerror(errno));
		free(bo_gem);
		return NULL;
	}
	bo_gem->bo.size = open_arg.size;
	bo_gem->bo.offset = 0;
	bo_gem->bo.virtual = NULL;
	bo_gem->bo.bufmgr = bufmgr;
	bo_gem->name = name;
	atomic_set(&bo_gem->refcount, 1);
	bo_gem->validate_index = -1;
	bo_gem->gem_handle = open_arg.handle;
	bo_gem->bo.handle = open_arg.handle;
	bo_gem->global_name = handle;
	bo_gem->reusable = false;

	VG_CLEAR(get_tiling);
	get_tiling.handle = bo_gem->gem_handle;
	ret = drmIoctl(bufmgr_gem->fd,
		       DRM_IOCTL_I915_GEM_GET_TILING,
		       &get_tiling);
	if (ret != 0) {
		drm_intel_gem_bo_unreference(&bo_gem->bo);
		return NULL;
	}
	bo_gem->tiling_mode = get_tiling.tiling_mode;
	bo_gem->swizzle_mode = get_tiling.swizzle_mode;
	/* XXX stride is unknown */
	drm_intel_bo_gem_set_in_aperture_size(bufmgr_gem, bo_gem);

	DRMINITLISTHEAD(&bo_gem->vma_list);
	DRMLISTADDTAIL(&bo_gem->name_list, &bufmgr_gem->named);
	DBG("bo_create_from_handle: %d (%s)\n", handle, bo_gem->name);

	return &bo_gem->bo;
}

static void
drm_intel_gem_bo_free(drm_intel_bo *bo)
{
	drm_intel_bufmgr_gem *bufmgr_gem = (drm_intel_bufmgr_gem *) bo->bufmgr;
	drm_intel_bo_gem *bo_gem = (drm_intel_bo_gem *) bo;
	struct drm_gem_close close;
	int ret;

	DRMLISTDEL(&bo_gem->vma_list);
	if (bo_gem->mem_virtual) {
		VG(VALGRIND_FREELIKE_BLOCK(bo_gem->mem_virtual, 0));
		munmap(bo_gem->mem_virtual, bo_gem->bo.size);
		bufmgr_gem->vma_count--;
	}
	if (bo_gem->gtt_virtual) {
		munmap(bo_gem->gtt_virtual, bo_gem->bo.size);
		bufmgr_gem->vma_count--;
	}

	/* Close this object */
	VG_CLEAR(close);
	close.handle = bo_gem->gem_handle;
	ret = drmIoctl(bufmgr_gem->fd, DRM_IOCTL_GEM_CLOSE, &close);
	if (ret != 0) {
		DBG("DRM_IOCTL_GEM_CLOSE %d failed (%s): %s\n",
		    bo_gem->gem_handle, bo_gem->name, strerror(errno));
	}
	free(bo_gem->aub_annotations);
	free(bo);
}

static void
drm_intel_gem_bo_mark_mmaps_incoherent(drm_intel_bo *bo)
{
#if HAVE_VALGRIND
	drm_intel_bo_gem *bo_gem = (drm_intel_bo_gem *) bo;

	if (bo_gem->mem_virtual)
		VALGRIND_MAKE_MEM_NOACCESS(bo_gem->mem_virtual, bo->size);

	if (bo_gem->gtt_virtual)
		VALGRIND_MAKE_MEM_NOACCESS(bo_gem->gtt_virtual, bo->size);
#endif
}

/** Frees all cached buffers significantly older than @time. */
static void
drm_intel_gem_cleanup_bo_cache(drm_intel_bufmgr_gem *bufmgr_gem, time_t time)
{
	int i;

	if (bufmgr_gem->time == time)
		return;

	for (i = 0; i < bufmgr_gem->num_buckets; i++) {
		struct drm_intel_gem_bo_bucket *bucket =
		    &bufmgr_gem->cache_bucket[i];

		while (!DRMLISTEMPTY(&bucket->head)) {
			drm_intel_bo_gem *bo_gem;

			bo_gem = DRMLISTENTRY(drm_intel_bo_gem,
					      bucket->head.next, head);
			if (time - bo_gem->free_time <= 1)
				break;

			DRMLISTDEL(&bo_gem->head);

			drm_intel_gem_bo_free(&bo_gem->bo);
		}
	}

	bufmgr_gem->time = time;
}

static void drm_intel_gem_bo_purge_vma_cache(drm_intel_bufmgr_gem *bufmgr_gem)
{
	int limit;

	DBG("%s: cached=%d, open=%d, limit=%d\n", __FUNCTION__,
	    bufmgr_gem->vma_count, bufmgr_gem->vma_open, bufmgr_gem->vma_max);

	if (bufmgr_gem->vma_max < 0)
		return;

	/* We may need to evict a few entries in order to create new mmaps */
	limit = bufmgr_gem->vma_max - 2*bufmgr_gem->vma_open;
	if (limit < 0)
		limit = 0;

	while (bufmgr_gem->vma_count > limit) {
		drm_intel_bo_gem *bo_gem;

		bo_gem = DRMLISTENTRY(drm_intel_bo_gem,
				      bufmgr_gem->vma_cache.next,
				      vma_list);
		assert(bo_gem->map_count == 0);
		DRMLISTDELINIT(&bo_gem->vma_list);

		if (bo_gem->mem_virtual) {
			munmap(bo_gem->mem_virtual, bo_gem->bo.size);
			bo_gem->mem_virtual = NULL;
			bufmgr_gem->vma_count--;
		}
		if (bo_gem->gtt_virtual) {
			munmap(bo_gem->gtt_virtual, bo_gem->bo.size);
			bo_gem->gtt_virtual = NULL;
			bufmgr_gem->vma_count--;
		}
	}
}

static void drm_intel_gem_bo_close_vma(drm_intel_bufmgr_gem *bufmgr_gem,
				       drm_intel_bo_gem *bo_gem)
{
	bufmgr_gem->vma_open--;
	DRMLISTADDTAIL(&bo_gem->vma_list, &bufmgr_gem->vma_cache);
	if (bo_gem->mem_virtual)
		bufmgr_gem->vma_count++;
	if (bo_gem->gtt_virtual)
		bufmgr_gem->vma_count++;
	drm_intel_gem_bo_purge_vma_cache(bufmgr_gem);
}

static void drm_intel_gem_bo_open_vma(drm_intel_bufmgr_gem *bufmgr_gem,
				      drm_intel_bo_gem *bo_gem)
{
	bufmgr_gem->vma_open++;
	DRMLISTDEL(&bo_gem->vma_list);
	if (bo_gem->mem_virtual)
		bufmgr_gem->vma_count--;
	if (bo_gem->gtt_virtual)
		bufmgr_gem->vma_count--;
	drm_intel_gem_bo_purge_vma_cache(bufmgr_gem);
}

static void
drm_intel_gem_bo_unreference_final(drm_intel_bo *bo, time_t time)
{
	drm_intel_bufmgr_gem *bufmgr_gem = (drm_intel_bufmgr_gem *) bo->bufmgr;
	drm_intel_bo_gem *bo_gem = (drm_intel_bo_gem *) bo;
	struct drm_intel_gem_bo_bucket *bucket;
	int i;

	/* Unreference all the target buffers */
	for (i = 0; i < bo_gem->reloc_count; i++) {
		if (bo_gem->reloc_target_info[i].bo != bo) {
			drm_intel_gem_bo_unreference_locked_timed(bo_gem->
								  reloc_target_info[i].bo,
								  time);
		}
	}
	bo_gem->reloc_count = 0;
	bo_gem->used_as_reloc_target = false;

	DBG("bo_unreference final: %d (%s)\n",
	    bo_gem->gem_handle, bo_gem->name);

	/* release memory associated with this object */
	if (bo_gem->reloc_target_info) {
		free(bo_gem->reloc_target_info);
		bo_gem->reloc_target_info = NULL;
	}
	if (bo_gem->relocs) {
		free(bo_gem->relocs);
		bo_gem->relocs = NULL;
	}

	/* Clear any left-over mappings */
	if (bo_gem->map_count) {
		DBG("bo freed with non-zero map-count %d\n", bo_gem->map_count);
		bo_gem->map_count = 0;
		drm_intel_gem_bo_close_vma(bufmgr_gem, bo_gem);
		drm_intel_gem_bo_mark_mmaps_incoherent(bo);
	}

	DRMLISTDEL(&bo_gem->name_list);

	bucket = drm_intel_gem_bo_bucket_for_size(bufmgr_gem, bo->size);
	/* Put the buffer into our internal cache for reuse if we can. */
	if (bufmgr_gem->bo_reuse && bo_gem->reusable && bucket != NULL &&
	    drm_intel_gem_bo_madvise_internal(bufmgr_gem, bo_gem,
					      I915_MADV_DONTNEED)) {
		bo_gem->free_time = time;

		bo_gem->name = NULL;
		bo_gem->validate_index = -1;

		DRMLISTADDTAIL(&bo_gem->head, &bucket->head);
	} else {
		drm_intel_gem_bo_free(bo);
	}
}

static void drm_intel_gem_bo_unreference_locked_timed(drm_intel_bo *bo,
						      time_t time)
{
	drm_intel_bo_gem *bo_gem = (drm_intel_bo_gem *) bo;

	assert(atomic_read(&bo_gem->refcount) > 0);
	if (atomic_dec_and_test(&bo_gem->refcount))
		drm_intel_gem_bo_unreference_final(bo, time);
}

static void drm_intel_gem_bo_unreference(drm_intel_bo *bo)
{
	drm_intel_bo_gem *bo_gem = (drm_intel_bo_gem *) bo;

	assert(atomic_read(&bo_gem->refcount) > 0);
	if (atomic_dec_and_test(&bo_gem->refcount)) {
		drm_intel_bufmgr_gem *bufmgr_gem =
		    (drm_intel_bufmgr_gem *) bo->bufmgr;
		struct timespec time;

		clock_gettime(CLOCK_MONOTONIC, &time);

		pthread_mutex_lock(&bufmgr_gem->lock);
		drm_intel_gem_bo_unreference_final(bo, time.tv_sec);
		drm_intel_gem_cleanup_bo_cache(bufmgr_gem, time.tv_sec);
		pthread_mutex_unlock(&bufmgr_gem->lock);
	}
}

static int drm_intel_gem_bo_map(drm_intel_bo *bo, int write_enable)
{
	drm_intel_bufmgr_gem *bufmgr_gem = (drm_intel_bufmgr_gem *) bo->bufmgr;
	drm_intel_bo_gem *bo_gem = (drm_intel_bo_gem *) bo;
	struct drm_i915_gem_set_domain set_domain;
	int ret;

	pthread_mutex_lock(&bufmgr_gem->lock);

	if (bo_gem->map_count++ == 0)
		drm_intel_gem_bo_open_vma(bufmgr_gem, bo_gem);

	if (!bo_gem->mem_virtual) {
		struct drm_i915_gem_mmap mmap_arg;

		DBG("bo_map: %d (%s), map_count=%d\n",
		    bo_gem->gem_handle, bo_gem->name, bo_gem->map_count);

		VG_CLEAR(mmap_arg);
		mmap_arg.handle = bo_gem->gem_handle;
		mmap_arg.offset = 0;
		mmap_arg.size = bo->size;
		ret = drmIoctl(bufmgr_gem->fd,
			       DRM_IOCTL_I915_GEM_MMAP,
			       &mmap_arg);
		if (ret != 0) {
			ret = -errno;
			DBG("%s:%d: Error mapping buffer %d (%s): %s .\n",
			    __FILE__, __LINE__, bo_gem->gem_handle,
			    bo_gem->name, strerror(errno));
			if (--bo_gem->map_count == 0)
				drm_intel_gem_bo_close_vma(bufmgr_gem, bo_gem);
			pthread_mutex_unlock(&bufmgr_gem->lock);
			return ret;
		}
		VG(VALGRIND_MALLOCLIKE_BLOCK(mmap_arg.addr_ptr, mmap_arg.size, 0, 1));
		bo_gem->mem_virtual = (void *)(uintptr_t) mmap_arg.addr_ptr;
	}
	DBG("bo_map: %d (%s) -> %p\n", bo_gem->gem_handle, bo_gem->name,
	    bo_gem->mem_virtual);
	bo->virtual = bo_gem->mem_virtual;

	VG_CLEAR(set_domain);
	set_domain.handle = bo_gem->gem_handle;
	set_domain.read_domains = I915_GEM_DOMAIN_CPU;
	if (write_enable)
		set_domain.write_domain = I915_GEM_DOMAIN_CPU;
	else
		set_domain.write_domain = 0;
	ret = drmIoctl(bufmgr_gem->fd,
		       DRM_IOCTL_I915_GEM_SET_DOMAIN,
		       &set_domain);
	if (ret != 0) {
		DBG("%s:%d: Error setting to CPU domain %d: %s\n",
		    __FILE__, __LINE__, bo_gem->gem_handle,
		    strerror(errno));
	}

	if (write_enable)
		bo_gem->mapped_cpu_write = true;

	drm_intel_gem_bo_mark_mmaps_incoherent(bo);
	VG(VALGRIND_MAKE_MEM_DEFINED(bo_gem->mem_virtual, bo->size));
	pthread_mutex_unlock(&bufmgr_gem->lock);

	return 0;
}

static int
map_gtt(drm_intel_bo *bo)
{
	drm_intel_bufmgr_gem *bufmgr_gem = (drm_intel_bufmgr_gem *) bo->bufmgr;
	drm_intel_bo_gem *bo_gem = (drm_intel_bo_gem *) bo;
	int ret;

	if (bo_gem->map_count++ == 0)
		drm_intel_gem_bo_open_vma(bufmgr_gem, bo_gem);

	/* Get a mapping of the buffer if we haven't before. */
	if (bo_gem->gtt_virtual == NULL) {
		struct drm_i915_gem_mmap_gtt mmap_arg;

		DBG("bo_map_gtt: mmap %d (%s), map_count=%d\n",
		    bo_gem->gem_handle, bo_gem->name, bo_gem->map_count);

		VG_CLEAR(mmap_arg);
		mmap_arg.handle = bo_gem->gem_handle;

		/* Get the fake offset back... */
		ret = drmIoctl(bufmgr_gem->fd,
			       DRM_IOCTL_I915_GEM_MMAP_GTT,
			       &mmap_arg);
		if (ret != 0) {
			ret = -errno;
			DBG("%s:%d: Error preparing buffer map %d (%s): %s .\n",
			    __FILE__, __LINE__,
			    bo_gem->gem_handle, bo_gem->name,
			    strerror(errno));
			if (--bo_gem->map_count == 0)
				drm_intel_gem_bo_close_vma(bufmgr_gem, bo_gem);
			return ret;
		}

		/* and mmap it */
		bo_gem->gtt_virtual = mmap(0, bo->size, PROT_READ | PROT_WRITE,
					   MAP_SHARED, bufmgr_gem->fd,
					   mmap_arg.offset);
		if (bo_gem->gtt_virtual == MAP_FAILED) {
			bo_gem->gtt_virtual = NULL;
			ret = -errno;
			DBG("%s:%d: Error mapping buffer %d (%s): %s .\n",
			    __FILE__, __LINE__,
			    bo_gem->gem_handle, bo_gem->name,
			    strerror(errno));
			if (--bo_gem->map_count == 0)
				drm_intel_gem_bo_close_vma(bufmgr_gem, bo_gem);
			return ret;
		}
	}

	bo->virtual = bo_gem->gtt_virtual;

	DBG("bo_map_gtt: %d (%s) -> %p\n", bo_gem->gem_handle, bo_gem->name,
	    bo_gem->gtt_virtual);

	return 0;
}

int drm_intel_gem_bo_map_gtt(drm_intel_bo *bo)
{
	drm_intel_bufmgr_gem *bufmgr_gem = (drm_intel_bufmgr_gem *) bo->bufmgr;
	drm_intel_bo_gem *bo_gem = (drm_intel_bo_gem *) bo;
	struct drm_i915_gem_set_domain set_domain;
	int ret;

	pthread_mutex_lock(&bufmgr_gem->lock);

	ret = map_gtt(bo);
	if (ret) {
		pthread_mutex_unlock(&bufmgr_gem->lock);
		return ret;
	}

	/* Now move it to the GTT domain so that the GPU and CPU
	 * caches are flushed and the GPU isn't actively using the
	 * buffer.
	 *
	 * The pagefault handler does this domain change for us when
	 * it has unbound the BO from the GTT, but it's up to us to
	 * tell it when we're about to use things if we had done
	 * rendering and it still happens to be bound to the GTT.
	 */
	VG_CLEAR(set_domain);
	set_domain.handle = bo_gem->gem_handle;
	set_domain.read_domains = I915_GEM_DOMAIN_GTT;
	set_domain.write_domain = I915_GEM_DOMAIN_GTT;
	ret = drmIoctl(bufmgr_gem->fd,
		       DRM_IOCTL_I915_GEM_SET_DOMAIN,
		       &set_domain);
	if (ret != 0) {
		DBG("%s:%d: Error setting domain %d: %s\n",
		    __FILE__, __LINE__, bo_gem->gem_handle,
		    strerror(errno));
	}

	drm_intel_gem_bo_mark_mmaps_incoherent(bo);
	VG(VALGRIND_MAKE_MEM_DEFINED(bo_gem->gtt_virtual, bo->size));
	pthread_mutex_unlock(&bufmgr_gem->lock);

	return 0;
}

/**
 * Performs a mapping of the buffer object like the normal GTT
 * mapping, but avoids waiting for the GPU to be done reading from or
 * rendering to the buffer.
 *
 * This is used in the implementation of GL_ARB_map_buffer_range: The
 * user asks to create a buffer, then does a mapping, fills some
 * space, runs a drawing command, then asks to map it again without
 * synchronizing because it guarantees that it won't write over the
 * data that the GPU is busy using (or, more specifically, that if it
 * does write over the data, it acknowledges that rendering is
 * undefined).
 */

int drm_intel_gem_bo_map_unsynchronized(drm_intel_bo *bo)
{
	drm_intel_bufmgr_gem *bufmgr_gem = (drm_intel_bufmgr_gem *) bo->bufmgr;
	drm_intel_bo_gem *bo_gem = (drm_intel_bo_gem *) bo;
	int ret;

	/* If the CPU cache isn't coherent with the GTT, then use a
	 * regular synchronized mapping.  The problem is that we don't
	 * track where the buffer was last used on the CPU side in
	 * terms of drm_intel_bo_map vs drm_intel_gem_bo_map_gtt, so
	 * we would potentially corrupt the buffer even when the user
	 * does reasonable things.
	 */
	if (!bufmgr_gem->has_llc)
		return drm_intel_gem_bo_map_gtt(bo);

	pthread_mutex_lock(&bufmgr_gem->lock);

	ret = map_gtt(bo);
	if (ret == 0) {
		drm_intel_gem_bo_mark_mmaps_incoherent(bo);
		VG(VALGRIND_MAKE_MEM_DEFINED(bo_gem->gtt_virtual, bo->size));
	}

	pthread_mutex_unlock(&bufmgr_gem->lock);

	return ret;
}

static int drm_intel_gem_bo_unmap(drm_intel_bo *bo)
{
	drm_intel_bufmgr_gem *bufmgr_gem = (drm_intel_bufmgr_gem *) bo->bufmgr;
	drm_intel_bo_gem *bo_gem = (drm_intel_bo_gem *) bo;
	int ret = 0;

	if (bo == NULL)
		return 0;

	pthread_mutex_lock(&bufmgr_gem->lock);

	if (bo_gem->map_count <= 0) {
		DBG("attempted to unmap an unmapped bo\n");
		pthread_mutex_unlock(&bufmgr_gem->lock);
		/* Preserve the old behaviour of just treating this as a
		 * no-op rather than reporting the error.
		 */
		return 0;
	}

	if (bo_gem->mapped_cpu_write) {
		struct drm_i915_gem_sw_finish sw_finish;

		/* Cause a flush to happen if the buffer's pinned for
		 * scanout, so the results show up in a timely manner.
		 * Unlike GTT set domains, this only does work if the
		 * buffer should be scanout-related.
		 */
		VG_CLEAR(sw_finish);
		sw_finish.handle = bo_gem->gem_handle;
		ret = drmIoctl(bufmgr_gem->fd,
			       DRM_IOCTL_I915_GEM_SW_FINISH,
			       &sw_finish);
		ret = ret == -1 ? -errno : 0;

		bo_gem->mapped_cpu_write = false;
	}

	/* We need to unmap after every innovation as we cannot track
	 * an open vma for every bo as that will exhaasut the system
	 * limits and cause later failures.
	 */
	if (--bo_gem->map_count == 0) {
		drm_intel_gem_bo_close_vma(bufmgr_gem, bo_gem);
		drm_intel_gem_bo_mark_mmaps_incoherent(bo);
		bo->virtual = NULL;
	}
	pthread_mutex_unlock(&bufmgr_gem->lock);

	return ret;
}

int drm_intel_gem_bo_unmap_gtt(drm_intel_bo *bo)
{
	return drm_intel_gem_bo_unmap(bo);
}

static int
drm_intel_gem_bo_subdata(drm_intel_bo *bo, unsigned long offset,
			 unsigned long size, const void *data)
{
	drm_intel_bufmgr_gem *bufmgr_gem = (drm_intel_bufmgr_gem *) bo->bufmgr;
	drm_intel_bo_gem *bo_gem = (drm_intel_bo_gem *) bo;
	struct drm_i915_gem_pwrite pwrite;
	int ret;

	VG_CLEAR(pwrite);
	pwrite.handle = bo_gem->gem_handle;
	pwrite.offset = offset;
	pwrite.size = size;
	pwrite.data_ptr = (uint64_t) (uintptr_t) data;
	ret = drmIoctl(bufmgr_gem->fd,
		       DRM_IOCTL_I915_GEM_PWRITE,
		       &pwrite);
	if (ret != 0) {
		ret = -errno;
		DBG("%s:%d: Error writing data to buffer %d: (%d %d) %s .\n",
		    __FILE__, __LINE__, bo_gem->gem_handle, (int)offset,
		    (int)size, strerror(errno));
	}

	return ret;
}

static int
drm_intel_gem_get_pipe_from_crtc_id(drm_intel_bufmgr *bufmgr, int crtc_id)
{
	drm_intel_bufmgr_gem *bufmgr_gem = (drm_intel_bufmgr_gem *) bufmgr;
	struct drm_i915_get_pipe_from_crtc_id get_pipe_from_crtc_id;
	int ret;

	VG_CLEAR(get_pipe_from_crtc_id);
	get_pipe_from_crtc_id.crtc_id = crtc_id;
	ret = drmIoctl(bufmgr_gem->fd,
		       DRM_IOCTL_I915_GET_PIPE_FROM_CRTC_ID,
		       &get_pipe_from_crtc_id);
	if (ret != 0) {
		/* We return -1 here to signal that we don't
		 * know which pipe is associated with this crtc.
		 * This lets the caller know that this information
		 * isn't available; using the wrong pipe for
		 * vblank waiting can cause the chipset to lock up
		 */
		return -1;
	}

	return get_pipe_from_crtc_id.pipe;
}

static int
drm_intel_gem_bo_get_subdata(drm_intel_bo *bo, unsigned long offset,
			     unsigned long size, void *data)
{
	drm_intel_bufmgr_gem *bufmgr_gem = (drm_intel_bufmgr_gem *) bo->bufmgr;
	drm_intel_bo_gem *bo_gem = (drm_intel_bo_gem *) bo;
	struct drm_i915_gem_pread pread;
	int ret;

	VG_CLEAR(pread);
	pread.handle = bo_gem->gem_handle;
	pread.offset = offset;
	pread.size = size;
	pread.data_ptr = (uint64_t) (uintptr_t) data;
	ret = drmIoctl(bufmgr_gem->fd,
		       DRM_IOCTL_I915_GEM_PREAD,
		       &pread);
	if (ret != 0) {
		ret = -errno;
		DBG("%s:%d: Error reading data from buffer %d: (%d %d) %s .\n",
		    __FILE__, __LINE__, bo_gem->gem_handle, (int)offset,
		    (int)size, strerror(errno));
	}

	return ret;
}

/** Waits for all GPU rendering with the object to have completed. */
static void
drm_intel_gem_bo_wait_rendering(drm_intel_bo *bo)
{
	drm_intel_gem_bo_start_gtt_access(bo, 1);
}

/**
 * Waits on a BO for the given amount of time.
 *
 * @bo: buffer object to wait for
 * @timeout_ns: amount of time to wait in nanoseconds.
 *   If value is less than 0, an infinite wait will occur.
 *
 * Returns 0 if the wait was successful ie. the last batch referencing the
 * object has completed within the allotted time. Otherwise some negative return
 * value describes the error. Of particular interest is -ETIME when the wait has
 * failed to yield the desired result.
 *
 * Similar to drm_intel_gem_bo_wait_rendering except a timeout parameter allows
 * the operation to give up after a certain amount of time. Another subtle
 * difference is the internal locking semantics are different (this variant does
 * not hold the lock for the duration of the wait). This makes the wait subject
 * to a larger userspace race window.
 *
 * The implementation shall wait until the object is no longer actively
 * referenced within a batch buffer at the time of the call. The wait will
 * not guarantee that the buffer is re-issued via another thread, or an flinked
 * handle. Userspace must make sure this race does not occur if such precision
 * is important.
 */
int drm_intel_gem_bo_wait(drm_intel_bo *bo, int64_t timeout_ns)
{
	drm_intel_bufmgr_gem *bufmgr_gem = (drm_intel_bufmgr_gem *) bo->bufmgr;
	drm_intel_bo_gem *bo_gem = (drm_intel_bo_gem *) bo;
	struct drm_i915_gem_wait wait;
	int ret;

	if (!bufmgr_gem->has_wait_timeout) {
		DBG("%s:%d: Timed wait is not supported. Falling back to "
		    "infinite wait\n", __FILE__, __LINE__);
		if (timeout_ns) {
			drm_intel_gem_bo_wait_rendering(bo);
			return 0;
		} else {
			return drm_intel_gem_bo_busy(bo) ? -ETIME : 0;
		}
	}

	wait.bo_handle = bo_gem->gem_handle;
	wait.timeout_ns = timeout_ns;
	wait.flags = 0;
	ret = drmIoctl(bufmgr_gem->fd, DRM_IOCTL_I915_GEM_WAIT, &wait);
	if (ret == -1)
		return -errno;

	return ret;
}

/**
 * Sets the object to the GTT read and possibly write domain, used by the X
 * 2D driver in the absence of kernel support to do drm_intel_gem_bo_map_gtt().
 *
 * In combination with drm_intel_gem_bo_pin() and manual fence management, we
 * can do tiled pixmaps this way.
 */
void
drm_intel_gem_bo_start_gtt_access(drm_intel_bo *bo, int write_enable)
{
	drm_intel_bufmgr_gem *bufmgr_gem = (drm_intel_bufmgr_gem *) bo->bufmgr;
	drm_intel_bo_gem *bo_gem = (drm_intel_bo_gem *) bo;
	struct drm_i915_gem_set_domain set_domain;
	int ret;

	VG_CLEAR(set_domain);
	set_domain.handle = bo_gem->gem_handle;
	set_domain.read_domains = I915_GEM_DOMAIN_GTT;
	set_domain.write_domain = write_enable ? I915_GEM_DOMAIN_GTT : 0;
	ret = drmIoctl(bufmgr_gem->fd,
		       DRM_IOCTL_I915_GEM_SET_DOMAIN,
		       &set_domain);
	if (ret != 0) {
		DBG("%s:%d: Error setting memory domains %d (%08x %08x): %s .\n",
		    __FILE__, __LINE__, bo_gem->gem_handle,
		    set_domain.read_domains, set_domain.write_domain,
		    strerror(errno));
	}
}

static void
drm_intel_bufmgr_gem_destroy(drm_intel_bufmgr *bufmgr)
{
	drm_intel_bufmgr_gem *bufmgr_gem = (drm_intel_bufmgr_gem *) bufmgr;
	int i;

	free(bufmgr_gem->exec2_objects);
	free(bufmgr_gem->exec_objects);
	free(bufmgr_gem->exec_bos);
	free(bufmgr_gem->aub_filename);

	pthread_mutex_destroy(&bufmgr_gem->lock);

	/* Free any cached buffer objects we were going to reuse */
	for (i = 0; i < bufmgr_gem->num_buckets; i++) {
		struct drm_intel_gem_bo_bucket *bucket =
		    &bufmgr_gem->cache_bucket[i];
		drm_intel_bo_gem *bo_gem;

		while (!DRMLISTEMPTY(&bucket->head)) {
			bo_gem = DRMLISTENTRY(drm_intel_bo_gem,
					      bucket->head.next, head);
			DRMLISTDEL(&bo_gem->head);

			drm_intel_gem_bo_free(&bo_gem->bo);
		}
	}

	free(bufmgr);
}

/**
 * Adds the target buffer to the validation list and adds the relocation
 * to the reloc_buffer's relocation list.
 *
 * The relocation entry at the given offset must already contain the
 * precomputed relocation value, because the kernel will optimize out
 * the relocation entry write when the buffer hasn't moved from the
 * last known offset in target_bo.
 */
static int
do_bo_emit_reloc(drm_intel_bo *bo, uint32_t offset,
		 drm_intel_bo *target_bo, uint32_t target_offset,
		 uint32_t read_domains, uint32_t write_domain,
		 bool need_fence)
{
	drm_intel_bufmgr_gem *bufmgr_gem = (drm_intel_bufmgr_gem *) bo->bufmgr;
	drm_intel_bo_gem *bo_gem = (drm_intel_bo_gem *) bo;
	drm_intel_bo_gem *target_bo_gem = (drm_intel_bo_gem *) target_bo;
	bool fenced_command;

	if (bo_gem->has_error)
		return -ENOMEM;

	if (target_bo_gem->has_error) {
		bo_gem->has_error = true;
		return -ENOMEM;
	}

	/* We never use HW fences for rendering on 965+ */
	if (bufmgr_gem->gen >= 4)
		need_fence = false;

	fenced_command = need_fence;
	if (target_bo_gem->tiling_mode == I915_TILING_NONE)
		need_fence = false;

	/* Create a new relocation list if needed */
	if (bo_gem->relocs == NULL && drm_intel_setup_reloc_list(bo))
		return -ENOMEM;

	/* Check overflow */
	assert(bo_gem->reloc_count < bufmgr_gem->max_relocs);

	/* Check args */
	assert(offset <= bo->size - 4);